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Overview 



The Talent Development High School model is an education reform initiative that aims to im- 
prove the academic achievement of students in large, nonselective, comprehensive high 
schools. In operation at 33 high schools in 12 states across the country, the approach encom- 
passes five main features: small learning communities, organized around interdisciplinary 
teacher teams that share the same students and have common daily planning time; curricula 
leading to advanced English and mathematics coursework; academic extra-help sessions; staff 
professional development strategies; and parent- and community-involvement in activities that 
foster students’ career and college development. 

MDRC is evaluating the model at the invitation of the organization that created it, the Center 
for Research on the Education of Students Placed At Risk (CRESPAR), based at The Johns 
Hopkins University. Funding for this report was provided, through CRESPAR, by the U.S. De- 
partment of Education’ s Institute for Education Sciences. 

The report describes the context in which Talent Development operates, details the model’s 
components, and documents its implementation in five high schools in a large, urban school 
district. It presents findings on Talent Development’s effects on student achievement during the 
first three years of program operation, focusing on impacts for ninth-graders. The analysis is 
based on an innovative quasi-experimental research methodology. 

Key Implementation Findings 

• The high schools in the study are characterized by low student engagement, poor prior 
preparation among entering ninth-graders, low ninth-grade promotion rates, and continued 
problems in the upper grades. 

• Each Talent Development high school focused its initial implementation on the ninth grade 
by creating small learning communities, enacting curricular reforms, and providing profes- 
sional development for teachers. The implementation process was supported by a team of 
Talent Development organizational facilitators and coaches. 

Key Impact Findings 

• For first-time ninth-grade students, Talent Development produced substantial gains in aca- 
demic course credits and promotion rates and modest improvements in attendance. The per- 
centage of ninth-graders completing a core academic curriculum increased from 43 percent 
on average before the implementation of Talent Development to 56 percent after imple- 
mentation began. This increase is about three times the level of increase in similar schools 
in the district. Promotion rates in the Talent Development schools increased by just over 6 
percentage points, while they fell by 4 percentage points in the comparison schools. 

• Improvements in ninth-grade course credits earned, promotion, and attendance were 
strongest in the first three schools to begin using Talent Development, and these schools 
sustained improvements into the second and third years of implementation. 

Because these preliminary results are based on a small sample of schools in a single school dis- 
trict at the early stages of implementation, they should be interpreted with caution. Still, the ini- 
tial evidence of Talent Development's capacity to keep ninth-grade students on track for 
graduation is encouraging. Future reports will track outcomes for up to five years, include 
analysis of students in all high school grades, and examine the extent to which the fidelity of 
program implementation varies across schools and over time. 
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Preface 



In recent years, there has been a renewed national focus on the problems of urban high 
schools, including students’ poor graduation rates and high course failure rates. While such 
problems are rooted in the often inadequate preparation that students receive in elementary 
schools, their devastating effects become most visible in the ninth grade, when students 
encounter tougher requirements for grade-level promotion. In troubled high schools, a large 
percentage of students are retained in the ninth grade and many leave school before being 
promoted to the upper grades. These two phenomena together result in a leak in the pipeline 
leading toward graduation — so much so that often a given urban high school’s ninth-grade 
class makes up close to half the population of the entire school. The problem is well 
documented, but there is still little rigorous evidence about interventions that successfully 
improve outcomes for ninth-grade students. 

The Talent Development High School model is part of a larger trend in reform, in 
which whole-school reform models aim to improve performance and engagement outcomes for 
students through the use of major changes to both the organizational structure and educational 
processes of high schools. The models that function in this way — broadly referred to as 
comprehensive school reform (CSR) models — are developed both nationally and locally, and 
they receive support from a combination of federal, state, and local funding as well as from 
private foundations. The U.S. Department of Education has developed an initiative to expand 
the use of the models: In 1998 it launched the Comprehensive School Reform Demonstration 
program with an appropriation of $145 million. Funding for the program grew to $308 million 
for fiscal year 2003. (A third of funded schools, which receive at least $50,000 each for three 
years, are middle or high schools.) 

Talent Development reflects many of the core underlying principles embedded in the 
CSR movement and has been a key target of federal resources aimed at expanding the use of 
CSR reforms in high schools. The findings in this report — which offers an initial assessment of 
the first and most intensive effort at scaling up the use of the Talent Development model — 
indicate that Talent Development helps ninth-grade students make progress toward high school 
graduation. This assessment is based on the use of a pathbreaking analytic method that relies on 
a combination of before-and-after and comparison-schools methods. 

While the findings may be seen as a promising start toward increasing graduation rates 
and improving the skills of high school graduates, some questions remain. It is not yet clear, for 
example, whether schools have the capacity to sustain the Talent Development reform, build on 
gains for ninth-grade students, and produce positive impacts for tenth-, eleventh-, and twelfth- 
grade students. Future work from this evaluation will include analysis of outcomes for students 
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in upper grades and, in some schools, the evaluation will determine Talent Development’s 
impact on graduation rates. It will also consider other student achievement outcomes, including 
performance on state assessments in reading and math. Finally, process research is under way 
that will help us learn how, beyond changes in curriculum and school structure, Talent 
Development made a difference. 



Gordon Berlin 
Executive Vice President 
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Executive Summary 



In many large, nonselective, urban high schools, less than half the students who enter the 
ninth grade graduate, and those who do often leave with weak academic skills and inadequate 
preparation for further education or the workforce. One of the factors creating such conditions is 
the quality of schooling that occurs in lower grades: The elementary and middle schools that feed 
into nonselective, urban high schools are not equipping their students with the reading, writing, 
mathematics, and study skills they need to succeed. As a result, ninth-graders experience high 
rates of course failure, and as many as half are not promoted on time to the tenth grade. Students 
retained in the ninth grade are much more likely to drop out of high school eventually. 

The Talent Development High School model is a comprehensive reform initiative 
designed to help transform the structure and curriculum of large high schools in urban districts, 
with the aim of improving students' levels of achievement and raising teachers’ and students’ 
expectations. Talent Development was created by practitioners and researchers at the Center for 
Research on the Education of Students Placed At Risk (CRESPAR), based at The Johns 
Hopkins University; the model operates in 33 schools nationwide. MDRC is conducting an 
independent, third-party evaluation of Talent Development, funded by the U.S. Department of 
Education’s Institute for Education Sciences through CRESPAR. The evaluation and 
CRESPAR' s efforts to expand the use of Talent Development are part of the U.S. Department 
of Education’s Comprehensive School Reform Demonstration (CSRD) program. 

This report examines 5 large, nonselective, comprehensive high schools that 
implemented Talent Development in a northeastern, urban school district that includes 22 such 
high schools. 1 The district is the locus of Talent Development’s initial and most extensive 
scaling-up effort. As a whole, it is considering a broad high school reform initiative based on the 
model’s underlying principles, and just since the beginning of this evaluation, two more of its 
high schools have begun implementing the model. 

The report describes the context in which the Talent Development high schools in the 
district operate, explains the model’s core components, and examines the initial implementation of 
those components in the district. Its main focus is an assessment of Talent Development’s impact 
on ninth-grade students in the five implementing high schools during the first three years of 
program operation (1998-1999, 1999-2000, 2000-2001). The findings provide information about 
the effectiveness of an early but intensive phase of Talent Development’s expansion and offer 
lessons about the potential of the ongoing work of Talent Development and related reform efforts. 



'in order to preserve the anonymity of the subjects in this study, this report refers to the participating 
school district as “the district.” 
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Key Implementation Findings 

With many of its schools exhibiting low student achievement and high dropout rates, 
the district has a great need for high school reform. At the same time, because it is subject to a 
shifting policy environment, leadership changes at the district and school levels, and high rates 
of teacher turnover, the district represents a challenging context for implementing and 
sustaining positive change. 

The high schools in the district face a number of specific challenges. Together, these 
add up to one of the most troubling problems confronting large, urban high schools: the degree 
to which students become disengaged and eventually drop out. While this process typically 
begins before students reach high school, its devastating effects are concentrated in the ninth- 
grade. 



• The nonselective high schools in the district are characterized by low 
student engagement, poor prior preparation among entering ninth- 
graders, low ninth-grade promotion rates, and continued problems in 
the upper grades. 

More than 75 percent of the students in the district’s nonselective schools entered the 
ninth grade with reading and math skills below grade level, and over 50 percent could be 
considered chronic absentees (students with attendance rates of 80 percent or lower). Moreover, 
fewer than two-thirds of the students who entered the ninth grade during the three school years 
before Talent Development’s implementation in the district were promoted to the tenth grade 
for the following school year. Less than 40 percent were on schedule to graduate four years after 
starting high school. For those who continued on to the upper grades, only about 10 percent 
performed at or above grade level on state standardized tests. 

• Talent Development’s initial scaling-up effort in the district focused on 
establishing and refining the ninth-grade components of the model. 

For the schools in the study, early implementation focused primarily on the ninth grade. 
During the first year of implementation, each of the five schools created a Ninth Grade Success 
Academy. Within each Success Academy are self-contained teams that are composed of at least 
4 teachers from several disciplines and 150-200 ninth-grade students. The teacher teams have 
common daily planning time, and students share several classes with the same peers. 

Also, during the first implementation year, the schools established extended-length 
class periods for the ninth-grade to facilitate the provision of a double mathematics courseload 
and a double reading/English courseload. These double-load courses are part of a standards- 
based curriculum that was put into place for ninth-graders and that calls for a strategic reading 
class followed by an English class, a transitional math class followed by algebra 1 , and a study 
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and life-skills class. In addition, implementation of the model included providing professional 
development for ninth-grade teachers, such as coaching, team-teaching, and regular seminars. 
Finally, the Talent Development model includes an after-hours program called Twilight School 
for students with serious attendance and discipline problems that constitute barriers to 
enrollment in the regular school program. 

Even though the initial focus of Talent Development in the district was on the ninth 
grade, CRESPAR is creating new curricula (including, for example, double courseloads of 
geometry and English) and professional development opportunities for teachers in the upper 
grades. Moreover, each of the Talent Development schools has established career-themed small 
learning communities for grades 10 through 12. 

Full implementation of the Talent Development High School model includes: 

• Reorganizing schools into small learning communities, including a Ninth 
Grade Success Academy, Career Academies for the upper grades, and an 
after-hours Twilight School. 

• Instituting a research-based curriculum designed to move all students toward 
advanced high school work in English and mathematics. 

• Offering recovery opportunities and extra help for students who need it. 

• Providing professional development opportunities for teachers and 
administrators to support implementation of the recommended reforms. 

• Creating parent- and community-involvement activities that encourage 
students’ career and college development. 

A key feature of the implementation process for Talent Development is the support 
provided to each school by an on-site organizational facilitator and a team of coaches who work 
daily with school leaders to support implementation of the model. In addition, a team of Talent 
Development curriculum developers and trainers are in frequent contact with the school-based 
facilitators and with key members of the school’s leadership and instructional teams. According 
to CRESPAR, implementation costs may range from $250 to $300 per student per year, which 
includes materials, technical assistance, and salaries for curriculum coaches and a full-time 
program facilitator. Funding for these expenses typically comes from federal CSRD grants, 
local school districts, and national or community-based foundations. 

As of the 2001-2002 school year, the last year for which student data are available for 
this report, none of the five schools in the study had reached full implementation. Two of the 
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schools had implemented Talent Development for three years, one school had implemented it 
for two years, and two schools were in their first year of implementation. 



Analytic Approach 

There are few rigorous studies of the effectiveness of comprehensive high school reform 
interventions. Because such reforms affect an entire school, a great challenge in evaluating their 
impact lies in comparing the performance of students in intervention schools with that of similar 
students in similar schools. In the parlance of evaluation research, rigorous impact studies need to 
rely on a valid counterfactual to estimate what would have occurred in the absence of the 
intervention being studied. The evaluation described in this report breaks new ground by using a 
combination of particularly strong quasi-experimental evaluation methods: interrupted time series 
analysis, value-added analysis, and hierarchical linear modeling. The evaluation builds on the 
strengths of each method to address the limitations that any one might have alone. 

The interrupted time series analysis compares student performance in Talent 
Development schools with the performance of similar students in the same schools prior to 
Talent Development implementation. The value-added analysis compares the historical patterns 
(and deviations from these patterns) for the Talent Development schools with those of similar 
non-Talent Development schools during the same period to account for other factors in the 
broader school district that may influence student performance. Together these analytic 
approaches, combined with hierarchical linear modeling and statistical controls for individual- 
level differences in student characteristics, provide a rigorous basis on which to make causal 
inferences about the effects of Talent Development on student performance. It should be noted, 
however, that even this combination of approaches may not control for all factors that may 
confound such inferences. For example, the analytic approach may not account for the school 
leadership’s motivation to undertake a school change process and the influence that that may 
have had on both school functioning and student achievement, even without the Talent 
Development components and supports. 



Key Impact Findings 

The impact analysis in this report focuses on outcomes for first-time ninth-graders — 
those whose records indicated that they were in the ninth grade in the spring of the year under 
study and in the eighth grade the previous spring. This group made up the majority of the ninth- 
grade class at each school in the study. Repeating ninth-graders (who were in the ninth grade in 
the spring of the year under study and in the ninth grade the previous spring) made up another 
segment of each ninth-grade class, but a primary goal of Talent Development is on-time 
promotion from the ninth grade to the tenth; the model therefore specifically targets students 
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entering the ninth grade for the first time. Although the model also makes an effort to promote 
repeating ninth-graders midyear, available data did not permit a thorough investigation of 
impacts for students repeating the ninth grade. 

• For first-time ninth-grade students, Talent Development produced 
substantial gains in academic course credits earned and promotion rates 
and modest improvements in attendance. 

Figure ES-1 provides a summary of the key impact findings. The solid bars represent 
the changes in key outcomes for the Talent Development schools between the baseline period 
and the follow-up period. The white bars represent changes in key outcomes for the comparison 
schools during the same period. The difference between the two bars represents the impact of 
Talent Development. 

A major goal for Talent Development is to get ninth-grade students to complete what 
could be considered a core academic curriculum: earning at least five credits during the 
school year, with three of those credits being in mathematics, English, and science. On average, 
in Talent Development schools, about 43 percent of first-time ninth-graders completed a core 
academic curriculum before the implementation of Talent Development, and an average of 56 
percent completed this combination of courses during the years after the model began to be 
implemented. This 13 percentage point increase is represented by the first bar in Figure ES-1. 
By contrast, the percentage of first-time ninth-graders who completed a core academic 
curriculum in non-Talent Development comparison schools increased by only 4 percentage 
points (from 46 to 50 percent) during the same period. The 9 percentage point difference 
(between the 13 percentage point improvement and the 4 percentage point improvement) 
represents the impact of Talent Development on first-time ninth-graders completing a core 
academic curriculum. 

Talent Development produced even larger impacts on the percentage of first-time ninth- 
grade students completing math courses. The program had about a 12 percentage point impact 
on the rate at which students completed any type of math course and about a 19 percentage 
point impact on the rate at which students completed an algebra course. At the same time, 
Talent Development had about a 9 percentage point impact on the percentage of students 
earning one or more English credits. 

Talent Development also improved the overall rate of promotion from the ninth grade 
to the tenth (including both on-time promotion and midyear promotion). Promotion rates in 
Talent Development schools rose by 6 percentage points after the program began 
implementation. During the same time period, these rates fell by 4 percentage points in the 
comparison schools. Thus, Talent Development produced about a 10 percentage point impact 
on promotion from the ninth grade to the tenth. Talent Development impacts on attendance rates 
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Figure ES.l 

Impacts on Key Outcomes for First-Time Ninth-Grade Students 
in Talent Development Schools and Non-Talent Development Schools 



Impact = 18.4*** 



Impact = 1 1.7*** 
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■ Talent Development Schools 
□ Non-Talent Development Schools 



Impact = 9.4*** 
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Core Academic Math Credit 

Curriculum Completed Earned 



Algebra Credit English Credit 
Earned Earned 



Promoted to the 
10th Grade 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 9th-grade students from five clusters. Each cluster consisted of a Talent Development school 
matched with a group of between two and five non-Talent Development schools. Students in the sample were included 
on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit 
during a given school year. First-time 9th-grade students were defined as students whose records indicate that they 
were in the 9th grade in the year under study and in the 8th grade in the previous year's administrative data file. 

Each bar represents the difference between the baseline average (the average over the three-year period prior to the 
initial implementation of Talent Development for a given cluster) and the average over the follow-up period (a one-, 
two-, or three-year period, depending on the cluster, after Talent Development was implemented). The impact at 
follow-up was calculated as the difference in deviations from the baseline average between Talent Development 
schools and non-Talent Development schools. 

A two-tailed t-test was applied to the difference between the baseline average and the average over the follow-up 
period and to the impact at follow-up. Standard errors and statistical significance levels are adjusted to account for 
cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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outcomes (not shown in the figure) were more modest but still statistically significant: about a 3 
percentage point impact in overall attendance rates and a 5 percentage point impact on regular 
attendance rates (defined as 90 percent or higher). 

In summary, the analyses indicate that students in each of the five Talent Development 
schools exhibited strong improvements on several measures of engagement and performance 
over their pre-Talent Development counterparts in the same schools. By contrast, students in the 
comparison schools showed smaller or even negative changes during the same period. This 
amounts to consistent and positive Talent Development impacts in almost every follow-up year 
for each of the five schools. Further analyses show that the positive impacts for course credits 
and promotion outcomes are apparent in the very first year of Talent Development 
implementation. These impacts tend to be sustained for those schools for which more than one 
year of follow-up data is available. 

• Improvements in the ninth-grade outcomes were strongest in the first 
three schools to begin using the model, and these schools sustained 
improvements into the second and third years of implementation. 

In this study, the “pioneer” schools — the three that began using Talent Development 
first — implemented and sustained the model with a process and a level of intensity that set 
them apart from the other two schools. While positive impacts were present in the first year of 
implementation for all schools, the largest improvements occurred in the pioneer schools, where 
positive impacts were sustained into the second and third years of implementation. 



Sizing Up the Results 

As stated earlier, as many as half of the ninth-graders in the district do not earn 
sufficient course credits to be promoted on time to the tenth grade. Further analysis indicates 
that the district’s rates of persistence to the twelfth grade are dramatically lower for students 
who have repeated the ninth grade. The impacts reported here are promising because they show 
that Talent Development has positive and significant impacts for increasing course completion 
and for promotion to the tenth grade. Thus, Talent Development helps keep ninth-grade students 
on-track for graduation, which is one of the most important goals of current high school reform 
efforts. Because successful completion of the ninth grade is a necessary but not sufficient step 
toward earning a high school diploma, future reports will look explicitly at Talent 
Development’ s impact on graduation rates. 
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Next Steps 

As the evaluation moves forward, it will investigate how variation in implementation 
across school sites affects Talent Development's capacity to improve student engagement and 
performance. Because the intended components of implementation are essentially the same at 
each school, it will be critical to learn more about the extent to which the quality of 
implementation varies across schools and over time. As the study moves forward, the analysis will 
focus as much on the questions about implementation as it does on the questions about impacts. 

It will be particularly important to determine whether the strong impacts are sustained 
in the pioneer schools and whether they also accme to the upper grades in those schools. In 
addition, the generalizability of the findings will be enhanced if the same types of impacts occur 
eventually in the later-implementing schools. This is a key issue, as the success of later- 
implementing schools are an indication of Talent Development’s capacity to scale-up in a large 
district and also to work with schools that may not have the same leadership and initiative as the 
“pioneer” schools who were first to embrace the reform. 

Subsequent reports will also track outcomes for up to five follow-up years and will 
include analyses of tenth-, eleventh-, and twelfth-grade students. For some schools, where 
sufficient follow-up data can be collected, it will be possible to determine whether 
improvements in ninth-grade promotion rates translate to increased graduation rates. In 
addition, the evaluation will examine other measures of student achievement to support the 
findings related to course credits earned. 
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Introduction 

The Talent Development High School model operates in schools that face serious prob- 
lems with student attendance, discipline, achievement scores, and dropout rates to help trans- 
form them structurally, instructionally, and comprehensively (across all grades and depart- 
ments). The model began its first and most ambitious scaling-up effort in the large urban school 
district that is the focus of this report. 1 The nonselective high schools in the district (which 
serves more than 200,000 students) are typical of the types of schools that Talent Development 
was specifically designed to help and in which Talent Development has been most widely im- 
plemented. 2 The organizational and curriculum-related changes that constitute Talent Develop- 
ment aim to establish a strong, positive school climate for learning, promote high standards for 
English and mathematics coursework for all students, and provide professional development 
systems to support implementation of the recommended reforms. Each of these objectives is 
part of a larger set of goals: to enhance students’ school attendance, improve student learning, 
and keep students on-course toward high school graduation. 

The Talent Development High School model was initiated in 1994 through a partner- 
ship between the Center for Research on the Education of Students Placed At Risk 
(CRESPAR), based at The Johns Hopkins University, and Patterson High School in Baltimore, 
Maryland. The model has expanded to 33 high schools in 12 states across the country, and 
CRESPAR is seeking ways to refine and expand it further. MDRC is conducting an independ- 
ent, third-party evaluation of the model, funded by the U.S. Department of Education through 
its Institute for Education Sciences, as part of the Comprehensive School Reform Demonstra- 
tion program. 2 

This report describes the context in which the Talent Development model operates and 
the model’s components. It also presents impact findings from Talent Development's first three 
years of operation in five high schools in the district, providing a preliminary assessment of the 
model’s impact on key outcomes for ninth-grade students, the group most intensively targeted 
during initial program implementation. The focus is on engagement and performance outcomes 
that are likely to be most relevant to the early phases of Talent Development's implementation. 



'in order to preserve the anonymity of the subjects in this study, this report refers to the participating 
school district as “the district” and uses pseudonyms for individual schools. 

"A Talent Development Middle School model also exists, but this report focuses on the high school 
model. Later reports will detail the implementation and impacts of the middle school model. 

For more information on the Comprehensive School Reform Demonstration program, see 
http://www.ed.gov/offices/OERI/csrrdp.html. 
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Specifically, the report examines impacts on three behavioral indicators: daily attendance, 
course-taking, and promotion rates. (While this report focuses on impacts for ninth-grade stu- 
dents in early implementation years, future reports will estimate Talent Development's impacts 
for students in grades 10 through 12, include up to five years of impacts estimates for some 
schools, present additional student achievement outcomes, and describe in greater detail the im- 
plementation process and the relevant changes in school functioning.) 

The report is organized into six sections: 

• The first section provides evidence of the problems faced by the high schools 
in the district (where the Talent Development High School model not only be- 
gan its most intensive work but also has been used most pervasively). 

• The second section describes the model, focusing on the features and com- 
ponents that aim to help high schools attack the problems they face as di- 
rectly and immediately as possible. 

• The third section describes the status of Talent Development implementation 
in the five high schools that are the subject of the impact analysis. 

• The fourth section outlines the evaluation’s analytic approach, detailing the 
comparative interrupted time series methodological design that was used to 
analyze the data obtained in the study. 

• The fifth section presents a discussion of preliminary results from analyses 
that were conducted on the five high schools for which data are available, de- 
scribing the estimated effect of Talent Development on ninth-graders’ atten- 
dance rates, promotion rates, and credits earned toward graduation. 

• The final section discusses conclusions that may be drawn from the analyses, 
attempts to put these findings in the context of other comprehensive school 
reform models, and highlights goals for future Talent Development research. 



The State of High Schools in the District 

High schools that serve high-poverty populations face overwhelming problems. The 
typical nonselective high school in a high-poverty neighborhood graduates less than half the 
students who enter the ninth grade, suffers from chronically low daily attendance rates — many 
students miss more than a month of schooling during a given school year — and provides an 
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often unsafe environment. 4 In addition, the relationship between students and faculty in such a 
school often lacks respect or caring. Such a weak environment for learning contributes to very 
low student achievement scores on tests in all core academic subjects. 5 

Following is an overview of the condition of nonselective, comprehensive secondary 
schools in the district under study, just before Talent Development began its work there. 

Large Minority Student Population 

Table 1 presents descriptive information for a group of 22 nonselective, comprehensive 
high schools in the district. The table captures the state of these schools over the three school 
years before Talent Development began scaling up its model in several of the high schools 
(1996-1997 through 1998-1999). 6 During this period, the 22 high schools enrolled approxi- 
mately 76 percent of the high school students in the district. 7 The table shows that approxi- 
mately 67 percent of the students were black, 17 percent were white, 1 1 percent were Hispanic, 
and the remaining 6 percent were of other races/ethnicities. About 1 1 percent of the students 
were classified for special education services. 

Low Student Engagement 

Among the most troubling problems in large urban high schools is the degree to which 
high school students become disengaged from school and eventually drop out. While this process 
typically begins before students reach high school, its devastating effects are concentrated in the 
ninth grade. The first column of Table 2 presents several indicators of the difficulty that ninth- 
graders in the district’s nonselective high schools had in progressing further. For example, among 
ninth-grade students, 41 percent were already overage for their grade, indicating that they had al- 
ready repeated a previous grade. In fact, at the time the data were collected, 27 percent of the 
ninth-graders were repeating the ninth grade. High levels of disengagement can be seen in the low 



4 Nonselective schools typically enroll students from a nearby neighborhood and do not require them to 
meet academic or other performance standards for admission. 

5 Legters, Balfanz, Jordan, and McPartland, 2002, pp. 4-13; Olson and Jerald, 1998. 

6 Note that the sample of sUidents that is the basis for the information in Table 1 and Table 2 excludes a 
substantial number of students who may have been listed in the district administrative records but who had not 
attempted any course credits. These tended to be students who dropped out of high school just prior to or dur- 
ing a given school year. Such students could not be included in the analysis because the district did not provide 
consistent information about them for the years included in this sUtdy. 

7 Approximately 19 percent of the district’s ninth- through twelfth-grade students attended selective high 
schools that admitted students on the basis of prior academic performance. Another 5 percent were enrolled in 
alternative schools or schools serving students with special needs. 
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Talent Development Evaluation 
Table 1 



Characteristics of Nonselective High Schools in 
a Large, Urban School District, 

School Years 1996-1997 Through 1998-1999 



Characteristic 




Average number of students 


1,821 


Average number of students per grade 


9th grade 


667 


10th grade 


483 


1 1th grade 


345 


1 2th grade 


325 


Race/ethnicity (%) 


Black 


67.2 


White 


16.7 


Hispanic 


10.6 


Other 


5.6 


Gender (%) 


Male 


51.4 


Female 


48.6 


Classified for special education (%) 


11.4 


Eligible for free/reduced lunch 3 (%) 


68.9 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes students from 22 nonselective, comprehensive high schools. Students in the sample were 
included on the district's transcript and attendance records. The sample excludes students who did not attempt at 
least one credit during a given school year. 

Calculated from the Common Core of Data for the 1999-2000 school year, provided by the U.S. Department of 
Education's National Center for Education Statistics. The figure shown represents the percentage of students who 
are eligible under the National School Lunch Act to participate in the federal free/reduced-price lunch program. 



attendance rates and high degree of chronic absenteeism (defined as having attendance rates of 
80 percent or lower). The table shows that nearly 58 percent of ninth-graders could be classified as 
chronic absentees (missing a total of more than seven weeks of school or an average of one day 
each week), while only 23 percent had attendance rates of 90 percent or higher. 

Poor Prior Preparation 

The typical ninth-grade student entered the district’s nonselective high schools with 
relatively low levels of reading and math skills. Based on SAT-9 achievement test scores from 
the eighth grade, the average ninth-grade student entered high school scoring at the 37th Normal 
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Talent Development Evaluation 
Table 2 

Characteristics of Students in Nonselective High Schools in 
a Large, Urban School District, 

School Years 1996-1997 Through 1998-1999 



Characteristic 


9th Grade 


10th- 12th Grade 


Overage for grade 3 (%) 


40.8 


35.4 


Currently repeating grade b (%) 


26.6 


11.4 


8th-grade SAT-9 test scores 
Reading Comprehension 

National Curve Equivalent (NCE) Score 


36.6 


n/a 


Percentage scoring above grade level 


23.4 


n/a 


Math total 


National Curve Equivalent (NCE) Score 


35.3 


n/a 


Percentage scoring above grade level 


16.9 


n/a 


llth-grade state assessment scores 0 
Reading Comprehension 

National Curve Equivalent (NCE) Score 


n/a 


27.0 


Percentage scoring above grade level 


n/a 


10.2 


Math total 


National Curve Equivalent (NCE) Score 


n/a 


26.9 


Percentage scoring above grade level 


n/a 


8.6 


Attendance rate d (%) 


71.3 


77.1 


Students with an attendance rate of: (%) 


90% or higher 


23.3 


28.6 


80% or lower 


57.5 


47.3 


Earned 4 or more course credits for the year 0 (%) 


58.2 


76.6 


Promoted to 10th grade on time 1 (%) 


61.0 


n/a 


Promoted to 12th grade on time 8 (%) 


40.7 


n/a 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes students from 22 nonselective, comprehensive high schools. Students in the sample were 
included on the district's transcript and attendance records. The sample excludes students who did not attempt at 
least one credit during a given school year. 

a Typically, students who are overage for grade were retained in the current grade or a prior one. “Overage for 
grade” means a student turned 15 before the start of the 9th grade, 16 before the start of the 10th grade, 17 before 
the start of the 1 1th grade, or 18 before the start of the 12th grade. 
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Table 2 (continued) 



b Students were defined as repeating a grade if the district's administrative records indicated that they were 
enrolled in the same grade for both the current and the previous year. 
c State standards assessment test scores were available only for the 1 1th grade. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the 
total number of days the student was enrolled in a given year. 

e Until the 1998-1999 school year, students in the district were required to earn four course credits in order to be 
promoted. 

r For the purposes of this analysis, 9th-grade students were considered to have been promoted to the 10th grade 
on time if they were listed as lOth-graders in the next year's administrative data file. Discrepancies between the 
percentage of students meeting various promotion requirements and the promotion rate may be caused by students 
earning some credits in previous years, incomplete course-detail records, or inconsistent application of the 
promotion requirements. 

s For the purposes of this analysis, 9th-grade students were considered to have been promoted on time to the 12th 
grade if they were listed as 12th-graders in the administrative data file three years after having finished the 9th 
grade. 



Curve Equivalent (NCE) in reading comprehension and at the 35th NCE in math computation 
and problem solving (the national average is the 50th NCE for both tests). 8 Among ninth-grade 
students, 23 percent entered high school with test scores at or above grade level in reading and 
1 7 percent entered with scores at or above grade level in math. 

Low Ninth-Grade Promotion and Graduation Rates 

Finally, Table 2 shows the rate at which ninth-grade students from the district’s schools 
were actually promoted to the tenth grade and were eventually promoted to the twelfth grade on 
time for their scheduled graduations. In all, fewer than two-thirds (61 percent) of ninth-graders 
were promoted to the tenth grade for the following school year. This percentage masks a more 
troubling pattern among students who had already repeated the ninth grade. Though not shown 
in the table, further analysis indicates that students repeating the ninth grade were much less 
likely to be promoted than students who were in the ninth grade for the first time. In all, 49 per- 
cent of the repeating ninth-graders were promoted to the tenth grade for the following year, 
compared with 66 percent of the first-time ninth-graders. 



8 

The Normal Curve Equivalent (NCE) is a way of measuring where a student falls along the normal curve. 
The normalized test score, which ranges from 1 to 99 with a mean of 50, allows for comparison across tests 
and subjects. Unlike percentile rank scores, the NCE measurement has an equal interval between scores, which 
means that NCE scores can be averaged to allow for comparison of groups of students or schools. 
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Only 41 percent of the ninth-graders from the 1996-1997 through 1998-1999 school years 
were enrolled in a district public high school as twelfth-graders three years later, when they were 
scheduled to graduate. Again, the rates for those repeating the ninth grade (not shown in the table) 
were dramatically lower (16 percent) than those for first-time ninth-graders (51 percent). 9 

Continued Problems in Upper Grades 

The second column of Table 2 presents information about the attendance, test score aver- 
ages, and promotion status of tenth- through twelfth-grade students. In short, the data in Table 2 
suggest that even though these students were able to progress beyond the ninth grade, their suc- 
cess in high school was by no means guaranteed. Over 10 percent were repeating their current 
grade, and attendance rates averaged 77 percent (indicating that a typical student was absent for an 
average of 45 days during the year — nearly the equivalent of a full marking period). Among stu- 
dents who reached the eleventh grade and took the state’s standardized assessment tests, only 10 
percent performed at or above grade level for both reading comprehension and math. 

In short, the information in Table 2 indicates that ninth-grade students in the nonselec- 
tive, comprehensive high schools in the district are at high risk of leaving school with very low 
levels of reading and math skills. 10 The vast majority of entering students appear to be unlikely 
to earn a high school diploma. Talent Development is designed specifically to intervene in such 
situations and to focus as directly and intensively as possible on many of the root causes of 
these problems, which begin in the ninth grade and continue throughout the upper grades. 



How Talent Development Aims to Improve High Schools 

As explained briefly earlier, Talent Development was conceived as a comprehensive 
paradigm for school reform, which asserts that all children can learn and must do so in an aca- 
demic setting that is demanding and that expresses high expectations. Wade Boykin, 
CRESPAR’s co-director, writes: 

This is a reachable goal for schools if they are committed to providing the 
appropriate support and the appropriate structure, assistance, and conditions 



9 The analysis is not able to track students who leave the district’s public schools. Students who are no 
longer enrolled in a district public high school may have dropped out of school or may have enrolled in a pub- 
lic high school in another district or in a private school. 

10 The averages presented in Table 1 and Table 2 mask the variation among high schools in the district, 
some of which serve somewhat more affluent communities and enable somewhat higher percentages of their 
students to make adequate progress through school. Talent Development aims specifically to serve students in 
the lowest-performing schools, many of which fall well below the averages presented in Table 1 and Table 2. 
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for learning. It is attainable if we change the traditional practice of schooling 
from classifying, sorting, and weeding out of students, to maximizing every 
child’s potential for academic development. 1 1 

From the beginning, Talent Development’s central goals have been to help transform 
urban high schools into solid learning institutions that establish a strong, positive school climate 
for learning, promote high standards for English and mathematics coursework for all students, 
and provide professional development systems to support implementation of the recommended 
reforms. To reach these goals, Talent Development aims to solve five problems common to 
many urban high schools: low student engagement, poor prior preparation, low ninth-grade pro- 
motion rates, low graduation rates, and isolation from the community. 12 

The Talent Development High School Model: Conceptual Framework 

As part of its early work with the CRESPAR team that is refining and scaling up the 
Talent Development model, MDRC constructed a conceptual framework describing Talent De- 
velopment’s theory of change. This research-based theory of change identifies the problems 
Talent Development attempts to address, specifies the model’s core components, and defines 
the key short-term and long-term goals it aspires to accomplish. Most importantly, the theory of 
change attempts to make explicit the pathways through which the core components of the 
model are intended to improve school functioning and, ultimately, student outcomes. 

Figure 1 presents a simplified version of this framework. The top row identifies ele- 
ments of local contexts that shape the ways the Talent Development approach has been adapted 
and sustained over time. These contextual elements include the policies and administrative prac- 
tices of states, school districts, and host schools, including funding and school finance struc- 
tures. They also include the characteristics of the local labor market, employers, postsecondary 
education systems, students, families, teachers, and local communities, and the existing organ- 
izational structures and curricula of high schools. The framework envisions these elements af- 
fecting four constructs: the implementation of the model’s structural elements, the resulting 
changes in supports and learning opportunities, the mediating outcomes, and student academic 
performance outcomes. The second row of the figure highlights several of the most difficult 
problems that high schools face. The next four rows lay out the four constructs that are affected 
by context and that outline the Talent Development reform and its intended effects: 



"Boykin, 2000, p. 7. 

"Jordan, McPartland, Legters, and Balfanz, 2000, p. 160. 
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Figure 1 

Simplified Conceptual Framework for the Talent Development High School Model 








• Structural elements: Describes the changes that Talent Development seeks 
to implement, including changes in schools’ organization, policies, curricu- 
lum content, resource allocations, and relationships with external entities. 

Talent Development encompasses five broad and interlocking elements: (1) 
reorganizing schools into small learning communities (including a Ninth 
Grade Success Academy, Career Academies for the upper grades, and a Twi- 
light School for students with serious attendance and discipline problems that 
constitute barriers to enrollment in the regular school program); (2) research- 
based curricula designed to move all students toward advanced coursework 
in English and mathematics; (3) recovery opportunities and extra help for 
students who need it; (4) staff professional development systems designed to 
support implementation of the recommended reforms; (5) parent and com- 
munity involvement activities that aim to encourage students’ career and col- 
lege development. These structural elements should be viewed as mecha- 
nisms that are mutually reinforcing and that offer direct and concrete ap- 
proaches to enhancing supports and learning opportunities. 

• Supports and learning opportunities: Include improvements in school cli- 
mate and functioning; positive changes in teacher and student behaviors, ex- 
periences, and expectations; and productive use of internal and external re- 
sources. As with the structural elements, the supports and learning opportuni- 
ties should be considered mutually reinforcing; together they are aimed at 
enhancing student performance through mediating outcomes. 

• Mediating outcomes: Encompass changes in students’ attitudes, levels of 
engagement, and sense of efficacy and competence. These mediating out- 
comes are direct antecedents to improvement in student performance, pro- 
gress through high school, and preparation for transitions to further education 
and employment. 

• Performance outcomes: Include positive changes in student achievement, 
progress toward graduation, and preparation for successful transitions to 
postsecondary education and employment. 

How the Model Addresses Key Problems: Components of Full 
Implementation 

Linkages among and between each stage in the conceptual framework illustrate the hy- 
pothesized pathways through which the Talent Development High School model is expected to 
affect student performance. In addition to representing the theory of change that drives the model, 
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the framework can also be used to outline the components of full implementation and to guide the 
measurement and analysis of the model's impacts and implementation. This section describes the 
components of full implementation; the following one outlines initial phases of implementation in 
the five schools in this study. (None of the schools has yet reached full implementation.) 

Key Problem: Anonymity 

A reason often given by students for dropping out of high school is that they feel dis- 
tance and estrangement from teachers and administrators . 13 A positive school climate, where 
students and adults know each other well and where adults express care and concern for stu- 
dents’ welfare, intellectual growth, and educational success, is a key motivational element in the 
learning process for adolescents . 14 The large size of comprehensive high schools often deper- 
sonalizes the school environment, preventing teachers from working in teams or developing an 
atmosphere conducive to learning . 15 Continuous changes in classroom composition and student 
peer groups also increase anonymity and diminish students’ sense of community. Students do 
not have a consistent group of teachers who are accountable for their success, and teachers do 
not have a chance to coordinate their coursework. To battle these factors, Talent Development 
uses small learning communities in an effort to build personalized relationships among and be- 
tween students and teachers. The small learning communities take the following forms: 

• Ninth Grade Success Academy. At the core of Talent Development restruc- 
turing is the Ninth Grade Success Academy, a self-contained school-within- 
a-school organized around interdisciplinary teacher teams that share the same 
students and have common daily planning time. Practices and offerings are 
designed to help ease students’ transition into high school, encourage good 
attendance, and promote positive learning behaviors. 

• Career Academies. For students in grades 10 through 12, Talent Develop- 
ment high schools are organized into Career Academies, which are self- 
contained groups with their own management and instructional staffs located 
in a separate part of school buildings. They each enroll 250 to 350 students 
and are organized around career themes. Career academies provide all stu- 
dents with a core college preparatory curriculum and work-based learning 
experiences supported by industry partners. 



13 Altenbaugh, 1998. 
l4 Wilson and Corbett, 1999. 

15 Sizer, 1984; Hill, Foster, and Gendler, 1990; Powell, Cohen, and Farrar, 1985. 
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• Advisors. Each student is assigned an advisor who serves as his or her advo- 
cate through the final three years of high school. 16 

Key Problem: Low Student Expectations 

Adolescent students become bored, and attendance suffers when students are not drawn 
to their classwork by the prospect of interesting and fulfilling activities or when they see no 
connections between learning tasks and their own interests and future goals. In traditional 
schools, the curriculum is usually separated into higher-level academic classes for college- 
bound students and lower-level academic and vocational classes for those presumed not to be 
college-bound. This separation often confines the teaching and learning process to the transfer 
of abstract knowledge from teachers to students or, in the case of vocational classes, to a narrow 
focus on specific job skills. For both groups, there are usually very few opportunities to explore 
how basic skills are actually applied outside the classroom. 17 

The Talent Development model involves organizational and instructional reforms that 
aim to fight apathy by connecting schoolwork to students’ backgrounds, interests, and goals, 
and by enlivening lessons and learning activities with interesting and challenging applications. 
Relevant reforms include: 

• Applying a meaningful focus to the curriculum. As part of the Talent De- 
velopment model, the Career Academies provide a curriculum that combines 
academic coursework necessary for graduation and for college admission 
(discussed further below) with a sequence of career-oriented courses and 
work-related awareness and development activities. Coursework that blends 
academic learning and career applications allows students to be involved in 
meaningful studies, tying occupational applications to core academic sub- 
jects. Each Career Academy has an Advisory Board to help design the cur- 
riculum, to provide internships or other experiential learning activities, and to 
assist teachers in blending career applications with core academic courses. 

• Providing a college preparatory sequence for all students. Central to the 
idea of high expectations for all students is a shift away from tracking and a 
shift toward college preparation for all students. The Talent Development 
small learning communities achieve this in the following ways: 



16 Center for Research on the Education of Students Placed at Risk, 2001a. 

17 See, for example. Resnick, 1987a; Raizen, 1989; Stasz et al., 1993; and Grubb, 1995. 
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• The Ninth Grade Success Academy includes doubling students’ English 
and mathematics courseloads. Although CRESPAR has created com- 
plete instructional programs only for the “catch-up” courses in reading 
and mathematics (discussed below), students are expected to take stan- 
dards-based courses in English 1, algebra 1, science, social studies, or 
history, and an elective. 

• In the upper grades, Career Academies have curriculum-planning com- 
mittees that coordinate with state and district mandates to provide all stu- 
dents with a rigorous course sequence of standards-based instruction. 
CRESPAR has developed a two-course geometry sequence for the tenth 
grade and has produced instructional guides, Talent Development Writ- 
ing and Student Team Literature , to be integrated into language arts 
courses. A typical Talent Development high school incorporates a four- 
year sequence in English and mathematics and offers science and social 
science courses at each grade level, along with career-pathway courses. 18 

• Creating extended class periods. Four 80- to 90-minute periods per day en- 
able schools to use a variety of learning activities that call for students to work 
individually and in cooperative teams on challenging and interesting topics. 

Key Problem: Poor Prior Preparation 

One of the greatest challenges secondary schools face is the wide diversity in the level 
and quality of preparation students receive prior to high school. The Talent Development model 
requires a common core curriculum for all students. It attempts to universalize standards-based 
education by providing increased academic learning time and significant recovery opportunities 
for struggling students. Elements of the high school model that attempt to address poor prior 
preparation and performance include: 

• Freshman Seminar. Offered during the first semester of the ninth grade, the 
Freshman Seminar is a study and life-skills course that provides in-depth les- 
sons using a variety of techniques to help students practice the studying, 
note-taking, time management, and social relations skills required in their 
academic subjects and lives outside of school. Course materials include a 
teacher’s manual of complete lesson plans, student workbooks, and readings. 



18 

Legters and Morrison, 1999, pp. 2.5 and 2.51; Center for Social Organization of Schools, 2002b. 
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• Block schedule. Talent Development works with schools to create a “4x4” 
extended-period block schedule (that is, four periods per day for four courses 
each semester) in order to offer “double doses” of English and math in the 
ninth and tenth grades. 

• Catch-up courses. A key feature of the doubling up of English and math 
courses are the ninth-grade catch-up courses that are offered for the first term 
to students with weak prior preparation. These include Strategic Reading and 
Transition to Advanced Mathematics. Strategic Reading is designed to meet 
the needs of students whose reading levels are two or more years below 
grade expectancy. The course focuses on listening skills, reading comprehen- 
sion, and guided practice, along with a Student Team Literature program, 
which uses partner discussion guides developed by CRESPAR to support 
cooperative learning in reading and language arts. 19 Students who take Stra- 
tegic Reading take an English 1 course in the second semester. Transition to 
Advanced Mathematics is designed to encourage students to recognize con- 
nections between mathematics and the world while preparing them for a rig- 
orous sequence of high school mathematics courses. Materials include peer- 
assisted starter activities, whole-class discovery lessons, and differentiated 
individual and small-group instruction guides for five multiweek units. Stu- 
dents who take Transition to Advanced Mathematics take an algebra 1 course 
in the second semester. 

• Extra help. Talent Development schools may offer additional courses for 
students who need extra support, including: (1) Literacy Lab, a course that 
offers extra language arts help, working in conjunction with Strategic Read- 
ing and allowing students to develop reading, writing, and listening skills in 
small groups. The use of educational software on computers and exposure to 
appropriate Internet-based texts are important elements of the lab. (2) Read- 
ing and Writing in Your Career, a balanced-literacy course with a writing 
component for tenth-grade students who need additional support in reading 
and writing. Lessons are designed to help students develop strategies and im- 
prove skills in an effort to enhance their success in all subject areas. 20 (3) Ge- 
ometry Foundations, a tenth-grade math catch-up course that prepares stu- 
dents for geometry. 



19 Center for Research on the Education of Students Placed at Risk, 2001b. 
20 Center for Social Organization of Schools, 2002b. 
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• Summer school and Saturday school. An on-site summer school and Sat- 
urday school allow students to make up failed course credits or receive extra 
academic help that they need for promotion. 

• Twilight School. An after-hours program is offered as an alternative to the 
regular school day for students who have serious attendance or discipline 
problems or who are returning to school from incarceration or suspension 
from another school. Instruction is offered in small classes, and extensive 
services are provided by guidance and support staff. 

Key Problem: Limited Capacity to Implement Comprehensive Reform 

Currently, most schools in highly stressed environments have little or no capacity to ad- 
dress the problems discussed so far: anonymity, low student expectations, and poor prior prepa- 
ration. Even with specific strategies such as the examples given, implementing a comprehensive 
set of organizational reforms that respond to these challenges requires that teachers and admin- 
istrators change their practices in fundamental ways. In light of this, two critical components of 
the Talent Development approach are sustained, multilayered, multiyear implementation sup- 
port and continuous professional development. These consist of: 

• A multiyear implementation plan. Talent Development staff work with 
schools to create, and to update and refine continually, a multiyear imple- 
mentation plan that defines a timeline for phasing-in the key components of 
the Talent Development model. CRESPAR provides schools with a mini- 
mum of two years of intensive implementation support for each component. 

• On-site facilitators and professional development. An on-site team of 
CRESPAR-trained facilitators supports the planning and implementation of 
reforms, organizes and delivers grade- and subject-specific professional de- 
velopment, and follows up workshops with in-class help. Teachers receive 
four tiers of continuous support: (1) Ongoing, curriculum-specific profes- 
sional development, focusing on modeling upcoming lessons, improving 
content knowledge, learning instructional strategies, and performing class- 
room management. For each course, there are two to three days of training in 
the summer and one two- to three-hour training session each month during 
the school year. (2) Weekly in-class implementation support from an experi- 
enced curriculum coach. In a given school, coaches typically are skilled 
school district teachers or administrators who are placed on special assign- 
ment to the Talent Development model or teachers or administrations from 
the school who are given release time. (3) School-based support from lead 
teachers who receive additional training in use of the curriculum and in in- 
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structional practices. (4) On-going technical assistance from CRESPAR- 
affiliated organizational facilitators, who coordinate components of Talent 
Development at the sites, including teacher coaching, curriculum materials 
and other resources, workshops, and student-teacher meetings. 

• A support team. Each school is assigned a support team of Talent Develop- 
ment trainers who maintain frequent contact with the school-based facilitators, 
principals, and key members of the school’s leadership and instructional teams. 

• The Talent Development network. Schools receive support from other 
schools in the Talent Development network. Since 1998, annual national 
conferences that convene Talent Development schools have allowed the 
schools to share their experiences and to learn from one another. There are 
also Talent Development institutes for principles, teacher coaches, and teach- 
ers. In addition, school-based organizational facilitators meet together several 
times throughout each year, coordinating with CRESPAR. 

Key Problem: Schools’ Isolation from Families, Communities, and Local 
Institutions 

Many high schools in the district are isolated from other institutions in their communi- 
ties and have very limited contact with students’ families (restricted to notification of severe 
disciplinary and academic problems). In addition, with few connections between schools and 
communities, particularly local employers, many students are inadequately informed about or 
prepared for the adult world and the world of work. 21 In short, for schools like the ones in this 
study, schooling can become a process of isolating students from the world they should be pre- 
paring to enter. Little effort is made to use the community as a resource for providing students 
with meaningful learning opportunities and a context for highlighting the potential relevance of 
what they are studying. 

To address this problem, through a partnership with the National Network of Partner- 
ship Schools (the Network, for short), Talent Development uses school-family-community 
partnerships. 22 A cornerstone of the Network is an “Action Team” approach, in which teachers, 
administrators, parents, community members, and students work together to design and imple- 
ment partnership activities that meet goals for student success. The goal is to enable families 
and communities to become informed about and involved in children’ s education and schools. 



21 

See, for example. Resnick, 1987b; Berryman and Bailey, 1992; Berryman, 1995. 

22 

Center for Social Organization of Schools, 2002a. 
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Facilitators from the Network work with Talent Development schools, employing a variety of 
strategies for increasing such involvement, including: 

• Offering programs to parents that provide parenting and child-rearing skills 
development 

• Communicating with families about school programs and student progress 

• Involving families as volunteers and audiences at the school 

• Involving families with their children in learning activities at home 

• Including families as participants in school decisions and governance 

• Collaborating with social service agencies to coordinate resources from the 
broader community for families and students 

The theory of change laid out in the preceeding subsections is the conceptual frame- 
work for the interacting components of the Talent Development High School model, illustrating 
how the model is expected to improve student outcomes. Talent Development is more prescrip- 
tive than other school reform approaches, but CRESPAR strives to balance high-fidelity im- 
plementation of core model components with the unique needs and circumstances of the par- 
ticipating schools and the capacities of teachers and administrators. Even if implemented inten- 
sively, a range of contextual factors at the school, district, or state level can enhance or limit the 
model’s capacity to make a positive difference for students. 



Context for Impacts: Implementation in Five High Schools 

This section describes the specific context for a preliminary assessment of Talent Devel- 
opment’s impacts on several of the outcomes illustrated in Figure 1. While the prior section de- 
scribed the model’s theory of change and the ideal components of full implementation, this one 
describes actual implementation in five schools (for a summary of the initial phases of implemen- 
tation, see Box 1). Because the evaluation does not yet include a systematic analysis of implemen- 
tation in these schools, the paragraphs that follow do not attempt to address the quality or intensity 
of implementation, but they give an overview of the implementation of key model components in 
the schools (for a summary of the components, see Box 2). 23 The goal of this overview is to shed 
light on hypotheses about when, where, and how one might expect Talent Development to begin 
making a difference for troubled high schools and the students they serve. 

" Future reports will investigate in more detail the quality and intensity of implementation of the Talent 
Development model in high schools in the district. 
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Box 1 



Initial Phases of Implementation in Five High Schools 

Planning Year 

• Key staff members attend series of Talent Development orientations. 

• Staff members develop concrete implementation timeline and strategy. 

• School leaders recruit teachers to be part of the Ninth Grade Success Academy team. 

• Faculty vote to adopt the model and build commitment for implementation. 

Implementation Year 1 

• Schools establish Ninth Grade Success Academy. 

• Staff members participate in professional development linked to the ninth-grade 
curriculum. 

• A full-time Talent Development facilitator works on-site at participating schools. 

• Schools establish after-hours Twihght School (with the exception of one school). 

• Small learning communities arc in place for some students in grades 10 through 12 
(one school estabhshed career-themed academies for all students). 

Implementation Years 2 and 3 

• Ah schools continue Ninth Grade Success Academy. 

• Ah schools continue staff professional development. 

• Talent Development facilitator continues to work at school sites full-time. 

• Four schools continue Twihght School program (one school still has yet to implement 
Twihght School). 

• Two schools have career-themed academies for all students in grades 10 through 12, 
and others continue to evolve plans to create career academies for all upper-grade 
students. 



As noted earlier, there were 22 nonselective, comprehensive high schools in the district 
in the 1998-1999 school year, the year prior to Talent Development's expansion in the district. 
By the 2001-2002 school year, five of these schools had begun using the Talent Development 
model to bring about comprehensive reform in their structures, climates, curricula, and instruc- 
tional strategies. Two of these high schools began implementing Talent Development in the 
1999-2000 school year; one began in 2000-2001; and the remaining two began in 2001-2002. 24 



24 One other high school began implementing some features of the Talent Development model in the 2000- 
2001 school year. This school is not included in the analysis because it discontinued implementation the fol- 
lowing year. Three additional high schools began implementation or planning for implementing during the 

(continued) 
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Box 2 

Core Components of Talent Development for Ninth-Graders 



Ninth Grade Success Academy 

• Small learning community 

• Often housed in separate wing or floor of school building 

• Interdisciplinary teaching teams 

• “4x4” extended-period block schedule (four 80- to 90-minute periods per day 
of four courses each semester) 

Standards-Based Curriculum 

• Double English courseload: Strategic Reading followed by English 1 

• Double mathematics courseload: Transition to Advanced Mathematics 
followed by Algebra 1 

• Freshman Seminar: study and life-skills course 

Professional Development Linked to Curriculum 

• Continuous, on-site teacher coaching by Talent Development facilitator 

• Team-teaching 

• Summer and monthly seminar's 

Twilight School 

• Alternative after-hours program 

• Targeted courses to get students back on track for graduation 

• Small classes and support services 



Table 3, presenting a timeline of implementation for the five schools, shows that, as of 2001- 
2002 (the most recent school year for which data are available), the schools under study had 
been working with Talent Development for at least one year and for as many as three years (not 
including the initial planning year). 



2002-2003 school year. These schools could not be included in the analysis for this report because follow-up 
data are only available through the 2001-2002 school year. 
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Talent Development Evaluation 
Table 3 

Implementation Timeline in Five High Schools 



School A 
School B 
School C 
School D 
School E 



1998-1999 




School Year 



1999-2000 



Implementation 
Year 1 



Implementation 
Year 1 




2000-2001 



Implementation 
Year 2 



Implementation 
Year 2 



Implementation 
Year 1 





2001-2002 



Implementation 
Year 3 



Implementation 
Year 3 



Implementation 
Year 2 



Implementation 
Year 1 



Implementation 
Year 1 



The three “pioneer” schools (the ones that began planning for Talent Development in 
the late 1990s) may have implemented and sustained Talent Development with a process and a 
level of intensity that differed from those of the other two schools. For example, the pioneer 
schools began at a point when some of the key components of the model were being refined or 
created, and, as a result, they likely experienced intense interaction with the model developers. 
Also, of the five schools, the pioneer schools had to shoulder the burden of creating a track re- 
cord of credibility for Talent Development in the district and were part of early efforts to adapt 
the model to local conditions and specific school circumstances. At the same time, these early- 
starting schools seem to have been the most entrepreneurial and the most willing and able to 
initiate a change process. The impact findings indicate that the later- starting schools seem to 
have been somewhat less well-equipped, at least initially, to take on the difficult task of com- 
prehensive school reform. 

Planning Year 

Each of the schools introduced the core elements of the Talent Development model in 
roughly the same sequence. They began with a planning year, working with Talent Develop- 
ment to build awareness among the faculty and staff about the reform model and its compo- 
nents. Awareness-building sessions took place at the schools and offered opportunities for 



- 20 - 







teachers to interact with model developers. Throughout the year, key staff members from the 
schools participated in a series of orientations and informational sessions on all aspects of the 
Talent Development model and attended Talent Development conferences and institutes. Dur- 
ing the planning year, schools engaged with CRESPAR to build faculty commitment to the 
model and to develop a concrete implementation timetable and strategy. 

In general, the schools did complete the most intensive aspects of their development 
and consensus-building process in the planning year. In fact, however, planning has actually 
been an ongoing process as school staff and Talent Development facilitators work to refine the 
Ninth Grade Success Academy and attempt to create or develop the Career Academies for the 
upper grades. 25 Even in cases where schools made a solid start on initial implementation, the 
staff members' level of fidelity and commitment to the various components needed to be rein- 
forced from year to year, as teachers or administrators transferred out of the building and as new 
staff arrived. Also, changes in district and state policies and administrative directives necessi- 
tated an ongoing process of refining or adapting established components after they were initially 
put in place. For example, with the increasingly high stakes of state assessments, state curricu- 
lum standards have been incorporated into the implementation. 



The First Three Years of Implementation: Putting the Components of 
Talent Development into Place 

Ninth Grade Success Academy 

Early implementation in the five schools focused primarily on the Ninth Grade Success 
Academy. During the first year of implementation, each of the schools created a Success Acad- 
emy, which is a small learning community for ninth-grade students and teachers. This involved 
reconfiguring master course schedules and moving all ninth-grade homerooms and classes to a 
separate section of the school building. There was an effort to provide teachers with shared 
planning time several times each week in order to facilitate team teaching. In the first year of 
implementation, each of the schools also established extended-period scheduling blocks for 
ninth-graders to facilitate the provision of double math and reading/English courses. As noted 
above, the transitional math course was put into place as a way to prepare students for algebra 1 
in the first semester and then move them through a full algebra 1 course in the second semester. 
The reading/English course was designed to help students develop strategic reading skills in the 
first semester and then move students through a full college preparatory English 1 course in the 
second semester. 

25 See Jordan, McPartland, Legters, and Balfanz, 2000, p. 171. 
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One important aspect of the Ninth Grade Success Academy is the establishment of 
separate teams for students who are repeating the ninth grade. This occurs because the needs of 
the two groups of ninth-graders who participate in Talent Development — first-time ninth- 
graders and repeating ninth-graders — are distinct. While students entering the ninth grade for 
the first time are attempting to navigate the difficult transition from a relatively small and sup- 
portive middle school to a much larger, less personalized, and more alienating high school, stu- 
dents who are repeating the ninth grade have already fallen behind the requirements needed to 
graduate and may be suffering the high levels of disengagement and alienation that can result 
from the lack of support they received in the first year in high school. At the five schools in the 
district, the teacher teams that formed specifically to work with repeating ninth-graders focused 
on helping students complete the courses necessary for promotion to the tenth grade (which can 
occur midyear). 

Twilight School 

For the district’ s repeating ninth-graders who did not earn midyear promotion, and for 
other groups of students, Twilight School is an option for earning credits toward promotion. All 
but one of the five schools (School E) established a Twilight School, which is an important 
component of Talent Development. Operated after regular school hours and on Saturdays, Twi- 
light School aims to help students catch up on the coursework and credits they need to meet 
promotion requirements and get back on track toward graduation. In the district, in addition to 
targeting repeating ninth-graders, the Twilight Schools sought to accommodate students in all 
grades who were returning to school after incarceration or extended absences for other reasons. 
They also helped students who were juggling work or parenting responsibilities that prevented 
them from attending school during regular hours. (In some cases, efforts were made to move 
students back into the regular school program and schedule after a period of time spent in Twi- 
light School. In other cases, students remained in the Twilight School program in order to meet 
their work or parenting responsibilities.) 

Upper Grades 

By the end of the third year of implementation, each of the five schools under study had 
incorporated the key elements of the Ninth Grade Success Academy and, with the exception of 
School E, had established a Twilight School, but implementation of the upper-grade restructuring 
and curriculum reforms was more limited. Each of the schools already operated Career Acad- 
emies or other theme-based small learning communities for at least some of the students in grades 
10 through 12. One of the schools (School E) had moved to a complete reorganization of its upper 
grades into Career Academies in the first year of implementation, and a second school (School B) 
moved to the same Academy arrangement it its second year of implementation. The other three 
schools were still developing plans and strategies to ensure that Career Academy options were 
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available for all students in grades 10 through 12 and that students were in a position to make in- 
formed decisions about which Academy would best fit their interests and aspirations. 

Finally, in the third year of implementation, one of the pioneer schools incorporated 
new tenth-grade extra-help courses, supported by a curriculum coach. These courses focused 
primarily on helping students meet the district’s new promotion requirements and prepare for 
the State Standards Assessment. 26 

Ongoing Model Development 

Talent Development continues to evolve and mature. CRESPAR and the support infra- 
structure that has emerged in the district continue to refine the existing components and to de- 
velop new features that are aimed at specific problems that the high schools face. For example, 
one high school has seen important benefits in the Ninth Grade Success Academy, particularly 
in creating a supportive set of small learning communities that provide a secure space for ninth- 
graders in its large campus and that allow smaller groups of students to “travel together” 
throughout the day. As a result, they are embarking on the creation of a version of Success 
Academy for tenth-graders, including small learning communities, a separate space in the build- 
ing, and block scheduling. In addition, CRESPAR and the model development team continue to 
create new curricula and professional development opportunities for teachers and administra- 
tors, particular for the upper grades. These include double courseloads of geometry and English 
in the tenth grade. The process of model maturation and local adaptation represents a critical 
context for interpreting and understanding the impact findings discussed later in the report. 



Analytic Approach and Data Sources 

In order to determine the net effect of Talent Development, it is necessary to compare 
the experiences of a group of students who were exposed to the model with a truly comparable 
group of students who were not. The ideal research situation would provide an absolutely reli- 
able estimate of the student performance levels that would have been observed in the absence of 
the intervention (that is, a counterfactual) and comparison of this estimate to actual student per- 
formance. Random assignment is the most reliable basis from which to construct estimates of 
the counterfactual, but, in this evaluation, it was not possible to randomly assign schools or stu- 
dents. Given this, the analytic approach that is used attempts to construct the best counterfactual 
possible in order to estimate the true impact of Talent Development. 

26 Though outside the scope of this report, it is important to note that, in the 2002-2003 school year, several 
of the schools had begun to implement new upper-grade Talent Development courses, and some began sys- 
tematically developing Career Academies. 
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Box 3 



Estimating Impacts with a Comparative 
Interrupted Time Series Design 

• For each outcome under study, in each Talent Development school, the outcome 
level is compared with the pattern in the same school before it implemented the 
reform (referred to in the report as the baseline average). 

• For each Talent Development school, the outcome levels in a group of com- 
parison schools — a set of schools in the same district with characteristics 
similar to those of the Talent Development schools — are compared with the 
baseline averages in these schools before the Talent Development school im- 
plemented the reform. 

• Differences between the deviations from the baseline averages in the Talent De- 
velopment schools and the deviations from the baseline averages in the compari- 
son schools are used to estimate the reform’s impact. 



In this report, impacts are measured using a comparative interrupted times series design 
(see Box 3). 27 This analytic approach combines the use of the interrupted time series analytic 
strategy with the use of the comparison schools analytic strategy to build on the strengths of 
each and to address the potential limitations of both. The comparative interrupted time series 
design compares deviations from the historical patterns for the Talent Development schools 
with deviations from the historical patterns for similar non-Talent Development high schools 
during the same period. Thus, impacts are defined as differences between Talent Development 
and non-Talent Development high schools in terms of their deviations from historical patterns 
in student outcomes. When combined with regression analysis to control for differences caused 
by individual student background characteristics and prior school experiences, the approach 
seeks to isolate Talent Development’s unique impact on student engagement and performance. 

Interrupted Time Series 

The interrupted time series component of the analytic strategy assesses the extent to 
which measures of engagement and performance for students in Talent Development schools 
differ from the engagement and performance for similar students in the same schools prior to 
Talent Development implementation. This provides an indication of whether the participating 

27 

A detailed description of the analytic approach is available online in this report’s Technical Resources 
(Unit 1: Analytic Appendix). See www.mdrc.org/publications/388/techresources.pdf. 
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high schools experienced a deviation from their historical patterns in student outcomes coinci- 
dent with the introduction of Talent Development. The projection of each school’s recent his- 
tory acts as the counterfactual. This is a particularly good counterfactual because, in the absence 
of the reform, many aspects of the school would be expected to stay the same (for example, stu- 
dents, faculty, school culture, neighborhood, and physical plant). Use of a historical pattern as 
the counterfactual has the potential to control for both measurable and unmeasurable character- 
istics of a school. 

However, the deviation from the baseline alone may not necessarily reflect the impact 
of Talent Development. Similar deviations from historical patterns could have been caused by 
district- wide policies or interventions that occurred at about the same time as Talent Develop- 
ment implementation. For example, while Talent Development was being scaled up, the district 
changed course requirements for grade-level promotion. Such a change may cause positive de- 
viations from baseline averages of course credits that were earned at schools in the district. An 
interrupted time series design would capture this improvement and ascribe it to Talent Devel- 
opment, but in reality Talent Development may have caused some, all, or none of this change in 
credits earned. In order to sort out what part of the deviation from baseline is caused by Talent 
Development, this study looks at similar high schools in the same district. 

Comparison Schools 

The use of comparison schools helps to account for other factors in the broader school 
district that may influence school functioning and student engagement and performance. The 
Talent Development schools are matched with sets of comparison non-Talent Development 
schools that are similar on several dimensions. Both sets of schools consist of nonselective 
comprehensive high schools in a single district. The schools are matched on racial/ethnic com- 
position, promotion rates of ninth-grade students, and the similarity of average test scores and 
attendance rates. Table 4 provides an indication of the extent to which the matching process re- 
sulted in a group of non-Talent Development schools that was comparable to the five Talent 
Development schools in the study. In general, the table indicates that the non-Talent Develop- 
ment schools are similar to the Talent Development schools in terms of race/ethnicity and prior 
test scores over the years leading up to Talent Development implementation. Talent Develop- 
ment schools exhibited lower attendance and promotion rates than their non-Talent Develop- 
ment counterparts. 

Measures of student achievement and engagement at the comparison schools should be 
a good estimate of what might have been observed in Talent Development schools in the ab- 
sence of the intervention; that is, they should be a good counterfactual. But differences between 
the Talent Development and comparison schools do not necessarily reflect only the impact of 
Talent Development. Some differences could be artifacts of differences in the prior trends in 
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Talent Development Evaluation 
Table 4 

Characteristics of Ninth-Grade Students in Talent Development 
and Non-Talent Development Comparison Schools, 
Averaged Over the Pre-Talent Development Baseline Period 



Characteristic 


Talent 

Development 

Schools 


Non-Talent 

Development 

Schools 


Difference 


Race/ethnicity (%) 


Black 


82.0 


85.4 


-3.4 


White 


0.9 


3.1 


-2.2 


Hispanic 


16.8 


9.6 


7.2 


Other 


0.3 


1.9 


-1.6 


Overage for grade 2 (%) 


49.6 


45.3 


4.3 


8th-grade SAT-9 test scores 
Reading Comprehension 

Normal Curve Equivalent (NCE) Score 


33.5 


35.7 


-2.2 


Math total 


Normal Curve Equivalent (NCE) Score 


32.3 


33.9 


-1.6 


Attendance rate b (%) 


67.0 


70.3 


-3.4 


Students with an attendance rate of: (%) 


90% or higher 


15.5 


19.7 


-4.2 


80% or lower 


67.7 


61.6 


6.2 


Earned 4 or more credits for the year 0 (%) 


50.9 


56.2 


-5.3 


Promoted to 10th grade 01 (%) 


52.3 


56.2 


-3.8 



(continued) 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and seven non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance records. The 
sample excludes students who did not attempt at least one credit during a given school year. 

Results in the non-Talent Development columns reflect averages across five clusters of non-Talent Development schools. 
Each cluster consisted of between two and five non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Estimates are not regression-adjusted for students' background characteristics or prior achievement. 

Numbers reflect averages over the three-year period prior to the initial implementation of Talent Development for a given 
cluster. 

“Typically, students who are overage for grade were retained in the current grade or a prior one. “Overage for grade" 
means a student turned 15 before the start of the 9th grade, 16 before the start of the 10th grade, 17 before the start of the 
1 1th grade, or 18 before the start of the 12th grade. 

b Attendance rates were calculated for each student by dividing the number of days the student was present by the total 
number of days the student was enrolled in a given school year. 
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Table 4 (continued) 

“Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order to be 
promoted. 

d For the purposes of this analysis, 9th-grade students were considered to have been promoted to the 1 Oth grade if they 
were listed as lOth-graders in the next year's administrative data file. Discrepancies between the percentage of students 
meeting various promotion requirements and the promotion rate may be caused by students earning some credits in previous 
years, incomplete course-detail records, or inconsistent application of the promotion requirements. 



student engagement and performance. For example, test scores for students in Talent Develop- 
ment schools were actually lower than test scores for students in non-Talent Development 
schools, and they improved only marginally after Talent Development began. At the same time, 
however, test scores for students attending similar schools in the district may actually have been 
declining over the same period. In this instance, Talent Development may have had a positive 
impact by preventing test scores from dropping rather than by improving the overall average. 
Such a pattern could only be observed by comparing an interrupted time series for both Talent 
Development and non-Talent Development comparison schools. 

Controlling for Changes in School Composition 

This analysis takes into account the fact that Talent Development schools (or compari- 
son schools) may experience a change in the composition of their student populations. For ex- 
ample, neighborhoods may undergo demographic changes or changes in geographic boundaries 
or rules governing school assignment. More important, Talent Development may cause a 
change in the student population by, for example, preventing students from dropping out of high 
school and perhaps keeping lower-performing students in school longer. In order to help ac- 
count for systematic changes in the characteristics of student cohorts over time, the analysis in- 
corporates individual student characteristics into the analytic model. 

Limitations 

The goal of the comparative interrupted time series approach is to make the causal in- 
ference that Talent Development produced the observed changes in student outcomes. In order 
to make this inference several assumptions are made. First, the projected baseline average for a 
given school is calculated; given the relative stability of most outcomes in the baseline period, 
this projection is a reliable predictor of future student outcomes. Second, it is assumed that dis- 
trict policies would affect students in Talent Development and non-Talent Development schools 
in similar ways. This assumption is supported because the use of comparison schools suffi- 
ciently captures district-wide events or interventions that may affect student outcomes and, as 
Table 4 shows, the comparison schools and Talent Development schools serve similar students 
with similar outcome levels. Third, it is assumed that variables included in multiple regression 
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adequately control for compositional changes in student characteristics. While there may be 
changes in student characteristics that cannot be measured in correlatation with student out- 
comes, the analysis accounts for shifts in student racial/ethnic composition and changes in stu- 
dents’ level of prior achievement. 

There may still be alternative explanations or other factors unrelated to Talent Devel- 
opment that contribute to the observed differences in student outcomes. Nevertheless, despite 
the limitations of the comparative interrupted time series approach, using it to pool impacts es- 
timates over several schools provides a valid estimate of the impact of Talent Development in 
high schools in the district. Thus, when discussing the findings, the report states that Talent De- 
velopment either increased or decreased the level of each student outcome. 

Because in the current study there are a limited number of schools and few years of fol- 
low-up data, these findings are considered preliminary. But the pooled estimates across the five 
schools discussed in the study maximize the reliability of the impact estimates. By pooling es- 
timates, the analysis can assess the likelihood that a nonzero impact was due to chance. In gen- 
eral, the larger the number of schools that exhibited a nonzero impact, the higher the likelihood 
that the analysis can detect real changes in student engagement and performance that were pro- 
duced by Talent Development. While the focus is on results from pooled estimates, results for 
individual schools are also discussed in order to highlight variation across sites; it should be 
noted, however, that these impacts are less reliable than the pooled estimates because estimates 
for any one school may be anomalous. 

Data Sources 

The primary sources of data for this analysis are individual students’ school records, 
which were obtained from the district. In general, administrative, attendance-related, and 
course-related information was obtained for all middle and high school students in the district at 
the end of each school year, from the 1995-1996 school year through the 2001-2002 school 
year. 28 Box 4 defines several key outcomes included in the analysis. 



28 r, 



This report’s online Analytic Appendix gives a brief overview of the types of information included in 
these datasets. See www.mdrc.org/pubhcations/388/techresources.pdf. 
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Box 4 

Definitions of Key Program Outcomes 



Attendance 

• Attendance rate: The total number of days a student was marked as present during a school 
year, divided by the total number of days the student was listed as enrolled. (These data were 
available consistently only for students who attended school for at least one day in the fourth 
marking period of the school year. The analysis was not able to include students who dropped 
out or left the district prior to that point.) 

• Chronic absenteeism: When a student had an attendance rate of 80 percent or lower for the 
year. 

• Regular attendance: When a student had an attendance rate of 90 percent or higher for the year. 

Course Credits Earned 

• Course credits earned: Indicators of whether a high school student earned course credits in se- 
lected subject areas, including English, mathematics, science, and algebra. 

• Earned four or more credits: A designation indicating that a student was given credit for having 
passed four or more courses in a given year. Until the 1998-1999 school year, ninth-grade stu- 
dents were required to earn at least four credits in order to be promoted to the tenth grade. Be- 
ginning in the 1998-1999 school year, minimum requirements for promotion included earning 
at least five credits during the year, with three of those credits awarded for completing one re- 
quired course in mathematics, one in English, and one in science. 

• Earned core academic credits: A designation indicating that a student completed a core aca- 
demic curriculum, earning at least five credits during the ninth grade, with three of those credits 
being in mathematics, English, and science. Because this designation relied on the district’s 
course-credit code, which does not distinguish between elective courses and required courses, 
the designation may include credits that students earned for elective courses or for required 
courses. Some elective courses did not meet the district’s new 1998-1999 promotion require- 
ments, but all of them counted toward total credits earned. 

Promotion 

• Promoted to the tenth grade: Classification for a student who was designated in the district’s 
administrative records as a ninth-grader in a given school year and as a tenth-grader in the fol- 
lowing school year. Note that this is not necessarily the same as indicating that a student met 
various criteria for promotion within one school year (such as earning the required number of 
course credits or completing a designated sequence). While the vast majority of students who 
were promoted according to this definition did in fact meet these requirements, some students 
may have completed some of the necessary requirements in a prior year. Therefore, the out- 
come includes both on-time promotion and midyear promotion for some students who needed 
more than one school year to earn promotion to the tenth grade. 
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Preliminary Impact Findings 

The preliminary impact findings for this study indicate that, for first-time ninth-grade 
students, Talent Development produced important gains in academic course credits earned and 
promotion outcomes and more modest improvements in attendance rates. Impacts for nearly all 
first-time ninth-grade student outcomes included in this analysis are positive and statistically 
significant, meaning that the improvement is not likely due to chance. 29 

Although by the end of the 2001-2002 school year some schools had implemented Talent 
Development longer than others, the five schools are equally weighted in the analyses. For 
Schools A and B, which implemented Talent Development in the 1999-2000 school year, the im- 
pacts reflect the average impact across three years of implementation; for School C, which im- 
plemented Talent Development in the 2000-2001 school year, the impacts reflect the average im- 
pact across two years of implementation; and for Schools D and E, which implemented Talent 
Development in the 2001-2002 school year, the impacts reflect the finding in only one year of 
implementation. The impacts for the five schools are averaged and presented as the impact of Tal- 
ent Development in the follow-up period, which refers to the period of implementation. Impact 
findings for first-time ninth-grade students are presented first, followed by impact findings for 
repeating ninth-grade students. 30 Implications of these findings are discussed in the next section. 

Impacts for First-Time Ninth-Grade Students 

First-time ninth-graders are students whose records indicate that they were in the ninth 
grade during the spring of the year under study and in the eighth grade during the previous 
spring. First-time ninth-graders made up the majority of the ninth-grade class at each school 
included in the study, and they are the primary focus of Talent Development’s Ninth Grade 



29 

Given that the primary focus of implementation has been on the ninth grade, it is reasonable to expect 
that the primary locus of potential impacts is likely to be among ninth-grade students. As a result, the findings 
discussed in this report focus on Talent Development’s impact on key outcomes for ninth-graders. These out- 
comes include attendance rates, course-taking patterns, and promotion rates. The analysis further investigates 
the extent to which Talent Development produced different effects for first-time ninth-graders and for repeating 
ninth-graders. This is because, as explained, some of the participating high schools differed in the way that they 
used Talent Development to meet the needs of first-time ninth-graders and the way that they used it to meet the 

needs of repeating ninth-graders. 

30 

The pattern of findings for all ninth-grade students in the analysis sample, which includes both first-time 
ninth-graders and repeating ninth-graders, was very similar to the pattern of findings for first-time ninth- 
graders, although somewhat weaker. A set of tables showing impacts for the full ninth-grade sample is avail- 
able in the online Technical Resources (Unit 2a: Supplementary Tables for All Ninth-Grade Students) at 
www.mdrc.org/publications/388/techresources.pdf. 



- 30 - 



Success Academy. At the schools in the district, first- time ninth-graders received the most in- 
tense treatment of Talent Development, and Talent Development seems to have had the greatest 
impact on this group. 

• Talent Development schools substantially increased the percentage of 
first-time ninth-graders completing a core academic curriculum associ- 
ated with the district’s tougher promotion requirements. 

Table 5 provides a summary of the impact findings for several measures of engagement 
and performance for first-time ninth-graders. A key indicator of Talent Development’s success 
is the percentage of students who completed a core academic curriculum, earning at least five 
credits during the year, with three of those credits being in mathematics, English, and science. 
As the table shows, an average of just over 43 percent of students in the Talent Development 
schools earned core academic credits in the baseline period, while 56 percent of students earned 
these credits in the follow-up period. The deviation from baseline average for Talent Develop- 
ment schools (labeled “difference” in Table 5) is about 13 percentage points. During the same 
time period, comparable non-Talent Development schools also increased the percentage of stu- 
dents earning core academic credits, in this case from about 46 percentage points to 5 1 percent- 
age points, an increase of just over 4 percentage points. The difference in the deviations from 
baseline for the Talent Development and non-Talent Development schools is the impact. In this 
illustration, the estimated impact of Talent Development on the percentage of students who 
earned core academic credits is an increase of about 9 percentage points. 31 

The Talent Development impact on earning core academic credits is further broken 
down in Table 6, which shows impacts for each school over time. The first three columns show 
impacts for each year of implementation. 32 The average impact over all years of implementation 
for each school is given in the fourth column. 33 The row labeled “all clusters” at the bottom of 



31 

It is important to note that, had the analysis simply looked at the average deviation from baseline for Tal- 
ent Development schools, the impact of Talent Development would have been overestimated to be about 13 
percentage points (a 12.8 percentage point deviation from baseline). Also, had the analysis simply compared 
the follow-up period averages for Talent Development and non-Talent Development schools, the impact of 
Talent Development would have been underestimated to be about 6 percentage points (the difference between 
56.0 percent and 50.5 percent). 

32 

"Again, because the date of implementation varied, the pioneer schools (Schools A, B, and C) have im- 
pacts for two or three follow-up years, while Schools D and E have impacts for only one follow-up year. 

33 

The average values in column 4 of Table 6 may not exactly equal the arithmetic average of columns 1, 2, 
and 3, because column 4 values were calculated by pooling student data across all years rather than by averag- 
ing annual means. 
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Talent Development Evaluation 
Table 5 



Impacts on Attendance, Course-Taking, and Promotion 
for First-Time Ninth-Grade Students 



Outcome 




Talent 

Development 

Schools 






Non-Talent 

Development 

Schools 




Impact at 
Follow-Up 


Impact 

Effect 

Size 


Baseline 


Follow-Up 


Difference 


Baseline 


Follow-Up 


Difference 


Attendance (%) 


















Attendance rate a 


72.7 


75.5 


2'8 *** 


75.4 


75.7 


0.2 


2 6 *** 


0.09 


Students with an attendance rate of: 


















90% or higher 


21.2 


26.6 


5 4 *** 


25.5 


25.7 


0.2 


5 2 *** 


0.13 


80% or lower 


57.6 


52.4 


-5.2 *** 


52.2 


50.5 


-1.7 * 


-3.5 


-0.07 


Course credits earned (%) 


















At least 1 English credit for the year 


65.4 


76.2 


10.8 *** 


67.3 


68.7 


1.4 


9 4 *** 


0.19 


At least 1 algebra credit for the year 


32.8 


60.2 


27.3 *** 


44.0 


52.8 


gg *** 


18.5 *** 


0.41 


At least 1 math credit for the year 


55.8 


72.1 


16.3 *** 


63.9 


68.5 


4(5 *** 


217 *** 


0.23 


4 or more credits for the year 3 


61.9 


69.8 


79 *** 


65.9 


66.2 


0.3 


7 g *** 


0.15 


Core academic credits c 


43.2 


56.0 


12.8 *** 


46.3 


50.5 


42 *** 


8.6 *** 


0.18 


Promotion (%) 


















Promoted to 10th grade 1 


60.7 


67.1 


6.5 


61.6 


57.7 


-3.9 * 


10.4 ** 


0.21 


(continued) 



SOURCE: MDRC calculations from individual students’ school records front a large, urban school district. 



NOTES: Sample includes 9th-grade students front five Talent Development high schools and seven non-Talent Development high schools. Students in the 
sample were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given 
school year. First-time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and in the 8 th 
grade in the previous year's administrative data file. 
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Table 5 (continued) 

Results in the Talent Development columns reflect averages across the five Talent Development schools. Results in the non-Talent Development columns 
reflect averages across five clusters of non-Talent Development schools. Each cluster consisted of between two and five non-Talent Development schools. 
Some non-Talent Development schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over the three-year period prior to the initial implementation of Talent Development for a given cluster. 

Numbers in the "Follow-Up” columns reflect averages over the entire follow-up period. The length of the follow-up period varies by cluster. Two of the 
earliest implementing clusters had a three-year follow-up period, one cluster had a two-year follow-up period, and two clusters had only a one-year follow-up 
period. 

Numbers in the "Difference” columns reflect the difference between the baseline average and the average over the follow-up period. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. 

The impact effect size was calculated by dividing the impact at follow-up by the standard deviation of the outcome for all 9th-grade students in the district's 
nonselective, comprehensive high schools from school years 1996-1997 through 1998-1999. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. (For the full 9th-grade sample, the analysis also controls for whether the student was repeating the 9th grade.) 

A two-tailed t-test was applied to the difference between the baseline average and the average over the follow-up period and the impact at follow-up, which 
is the difference in deviations from baseline averages between Talent Development and non-Talent Development schools. Standard errors and statistical 
significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

‘‘Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was 
enrolled in a given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order to be promoted. Beginning in the 1998- 
1999 school year, minimum requirements for promotion included earning at least five credits during the 9th grade, with three of those credits awarded for 
completing one required course in mathematics, one in English, and one in science. 

c “Core academic credits” is a designation indicating that a student completed a core academic curriculum, earning at least five credits during the 9th grade, 
with three of those credits being in mathematics, English, and science. Because this designation relied on the district's course-credit code, which does not 
distinguish between elective courses and required courses, the designation may include credits that students earned for elective courses or for required courses. 
Some elective courses did not meet the district's new 1998-1999 promotion requirements, but all of them counted toward total credits earned. 

d For the purposes of this analysis, 9th-grade students were considered to have been promoted if they were listed as lOth-graders in the next year's 
administrative data file. Discrepancies between the percentage of students meeting various promotion requirements and the promotion rate may be caused by 
students earning some credits in previous years, incomplete course-detail records, or inconsistent application of the promotion requirements. Due to data 
availability, this outcome includes data for only three of the five clusters of Talent Development and non-Talent Development schools. 




Talent Development Evaluation 
Table 6 

Impacts on the Percentage of First-Time Ninth-Grade Students 
Who Earned Core Academic Credits, by Cluster 







Impact at Follow-Up 




Cluster 


Year 1 


Year 2 


Year 3 


Average 2 


School A 


12.7 ** 


\-j 2 *** 


16.0 *** 


15.1 *** 


School B 


15.3 ** 


1.5 


12.0 


9.6 * 


School C 


15.1 * 


9.7 




12.4 ** 


School D 


2.4 






2.4 


School E 


3.5 






3.5 


All clusters 11 


9 g *** 


9.5 ** 


14 0 *** 


8.6 *** 


Effect size 


0.21 


0.20 


0.29 


0.18 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Clusters consisted of a Talent Development school matched with a group of between two and five non- 
Talent Development schools. The impacts for the Year 1, Year 2, Year 3, and Average columns were calculated as 
the difference between the deviation from the baseline for the Talent Development schools and the deviation from 
the baseline for the non-Talent Development schools. 

“Core academic credits” is a designation indicating that a student completed a core academic curriculum, earning 
at least five credits during the 9th grade, with three of those credits being in mathematics, English, and science. 
Because this designation relied on the district’s course-credit code, which does not distinguish between elective 
courses and required courses, the designation may include credits that students earned for elective courses or for 
required courses. Some elective courses did not meet the district’s new 1998-1999 promotion requirements, but all 
of them counted toward total credits earned. 

Estimates are regression-adjusted using ordinary least squares, controlling for students' background 
characteristics and prior achievement. 

A two-tailed t-test was applied to the difference between the baseline average and the average over the follow-up 
period and to the impact at follow-up, which is the difference in deviations from baseline averages between Talent 
Development and non-Talent Development schools. Standard errors and statistical significance levels are adjusted 
to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

“This column reflects the average of all students in the sample in the follow-up period. The length of the follow- 
up period varies by cluster. Two of the earliest implementing clusters had a three-year follow-up period, one cluster 
had a two-year follow-up period, and two clusters had only a one-year follow-up period. This column does not 
reflect the average of one-, two-, or three-year impacts, which are also included in the table. 

b All cluster averages are calculated vertically with the available data for each follow-up year and the overall 
average. 
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the table represents averages across the five schools for each implementation year. Table 6 illus- 
trates several patterns that occur throughout the findings for course credit and promotion out- 
comes. 34 First, while impacts for an individual school may be anomalous, there are consistently 
positive and statistically significant impacts for Schools A, B, and C, the first to implement the 
reform model. Second, positive impacts for course-credit and promotion outcomes are apparent 
in the very first year of implementation. Third, these impacts tend to be sustained for those 
schools for which more than one year of follow-up data are available. 

• Talent Development produced substantial gains in academic course 
credits earned by first-time ninth-grade students. The impact was espe- 
cially large for the percentage of students earning a credit in algebra. 

The Talent Development cuniculum includes the first-semester catch-up courses de- 
signed to help students complete English 1 and algebra 1 in the second semester of the ninth 
grade. As Table 5 shows, under “course credits earned,” Talent Development resulted in an 1 1 
percentage point increase in the percentage of first-time ninth-grade students who earned at least 
one credit in English, while non-Talent Development schools showed a percentage point increase 
of only about 1. This indicates that Talent Development had a 9 percentage point impact on the 
percentage of first-time ninth-graders earning at least one credit in English. The table also shows 
that the percentage of first-time ninth-graders earning at least one credit in algebra nearly doubled 
in Talent Development schools, by an increase of 27 percentage points. The non-Talent Devel- 
opment schools also show an increase in this outcome, but of only 9 percentage points. Therefore, 
the Talent Development impact on the percentage of first-time ninth-graders earning at least one 
credit in algebra is an increase of 19 percentage points. Talent Development also resulted in a 
positive impact of 12 percentage points on the percentage of first-time ninth-graders who earned 
at least one credit in math, and an impact of 8 percentage points on the percentage of first-time 
ninth-graders earning a total of four or more credits for the year. 

The increase in academic course completion throughout high schools in the district is 
not surprising. The district changed promotion requirements for ninth-graders the year before 
Talent Development was first implemented. Instead of simply earning four credits for promo- 
tion to the tenth grade, students were required to earn at least five course credits, including one 
credit in English, one in math, and one in science. Because of this factor, if the analysis calcu- 
lated simply the deviation from baseline for Talent Development schools, the impact of Talent 
Development may have been overestimated. The comparison schools help to account for dis- 
trict-wide policies or events that may affect student outcomes. In this case, the deviation from 



34 A full set of school-by-year outcome tables is available in the onhne Technical Resources (Unit 2: Sup- 
plementary Tables). See www.mdrc.org/publications/388/techresources.pdf. 
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baseline for non-Talent Development schools helps to capture the effect of a district-wide policy 
change. Subtracting this effect from the Talent Development deviation from baseline isolates 
the impact of Talent Development. 

• Talent Development schools improved the overall promotion rate to the 
tenth grade for first-time ninth-grade students. 

Table 5 shows two outcomes that are related to promotion: “core academic credits” and 
“promotion to 10th grade.” The first reflects the percentage of first-time ninth-grade students who 
completed a core academic curriculum by the end of the school year under study. Many of these 
students met the district’s new promotion requirements. 35 Talent Development schools increased 
the percentage of students who earned core academic credits by nearly 9 percentage points. 

The outcome indicating promotion to the tenth grade reflects the percentage of first-time 
ninth-grade students who reached the tenth grade by the end of the following school year. 36 The 
data indicate that Talent Development increased overall promotion to the tenth grade by 10 per- 
centage points. The reader should note that the outcome levels for both Talent Development 
schools and non-Talent Development schools are greater for this promotion outcome than for the 
core academic credits outcome. That is, in Talent Development schools in the follow-up period, 
56 percent of students completed a core academic curriculum, while 67 percent were actually 
promoted to the tenth grade. This reflects the fact that the second promotion outcome includes 
midyear promotion. (For example, some students did not complete all the courses required for 
promotion until midway through the following year.) Unfortunately, the data do not allow for stu- 
dents who were promoted on time and students who were promoted midyear to be directly identi- 
fied. However, from the available data, it is possible to estimate that about 16 percent of the first- 
time ninth-graders promoted in the Talent Development schools were promoted midyear, while 
about 18 percent of the first-time ninth-graders promoted in non-Talent Development schools 
were promoted midyear. 



35 .... . . 

“Core academic credits” is a designation indicating that a student completed a core academic curricu- 
lum, earning at least five credits during the ninth grade, with three of those credits being earned in mathemat- 
ics, English, and science. But because the “core academic credits” designation relied on the district’s course- 
credit code, which does not distinguish between elective courses and required courses, the designation may 
include credits that students earned for elective courses or for required courses. Some elective courses did not 
meet the district’s new 1998-1999 promotion requirements, but all of them counted toward total credits earned. 
(Beginning in the 1998-1999 school year, minimum requirements for promotion included earning at least five 
credits during the ninth grade, with three of those credits awarded for completing one required course in 
mathematics, one in English, and one in science.) 

36 Since data were obtained from the district at the end of each school year, it is not possible to determine 
directly which students achieved tenth-grade status on time and which were promoted midyear. 
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• Talent Development had a small but positive impact on attendance rates 
for first-time ninth-graders. 

While still statistically significant, the impacts on attendance outcomes were smaller 
than the impacts on course-credit and promotion outcomes. The average attendance outcome 
levels in the baseline period were similar for Talent Development and non-Talent Development 
schools. Average outcome levels remained relatively flat in the follow-up period for non-Talent 
Development schools, while attendance rates improved by 3 percentage points in Talent Devel- 
opment schools. Talent Development also increased the percentage of regular attenders, stu- 
dents with an attendance rate of 90 percent or better, by 5 percentage points. Attendance im- 
pacts varied across the five schools. As with the course-credit and promotion outcomes, the 
three pioneer schools exhibited greater impacts than the overall average (the reasons for this are 
discussed further below). 

• Improvements in the ninth-grade outcomes were strongest in the three 
schools that began using the model first, and the improvements were 
sustained into the second and third years of implementation. 

In general, cluster-by-cluster analyses indicate that each of the five Talent Development 
high schools exhibited positive deviations from its baseline average for course-credit outcomes 
during most of the years that it was implementing the model. 37 One school had negative devia- 
tions from the baseline for the English course -credit outcome, but its group of comparison 
schools averaged even larger negative deviations. This resulted in positive impacts for almost 
all of the individual follow-up years for course-credit outcomes. At the same time, the largest 
improvements occurred in Schools A, B, and C, the three pioneer schools. 

While it is not clear exactly why the pioneer schools did better, one hypothesis is that 
these schools were the most entrepreneurial of the five in the study and the most willing to initi- 
ate a change process. Despite incorporating the model at a point when some of the key compo- 
nents were being refined or created, in general the pioneer schools tended to have larger impacts 
on attendance and course completion than the full sample of five Talent Development schools 
in the district. The pioneers had positive attendance impacts (statistically significant in two 
schools), while the two schools that have most recently adopted Talent Development show 
small negative impacts (though not statistically significant ones) on attendance rate in the first 
year of implementation. 



37 

For further details, see the online Technical Resources (Unit 2: Supplementary Tables) at 
www.mdrc.org/publications/388/techresources.pdf. 
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Impacts for Schools A, B, and C were sustained during the second and third years of 
implementation, although the effects on attendance and course completion were somewhat 
smaller in the second year. The process and intensity of implementing the Talent Development 
model varied across the schools and over time. Teacher mobility and leadership turnover may 
have hindered the progression of implementation. Adding new components may have distracted 
from the ones already in place, because of limitations on school and Talent Development staff 
and resources. At the same time, however, it is likely that the structure and quality of the Ninth 
Grade Success Academies tended to improve over time as the scheduling challenges were ad- 
dressed and as teachers and administrators became more comfortable and familiar with the in- 
tended benefits of the small learning communities, the Freshman Seminar, and the extended 
class periods. 

It should be noted that, for most outcomes, the Talent Development baseline averages 
were lower than the baseline averages for the non-Talent Development comparison schools, yet 
the averages for the follow-up period were either the same or somewhat higher for the non- 
Talent Development schools. Thus, if one were to compare the Talent Development schools 
with the non-Talent Development schools only on the basis of the follow-up period averages, 
the program would appear to be making little or no difference. The differences in deviations 
from the baseline average, however, show that the model was able to move some of the most 
troubled schools in the district up to or beyond the levels of better-performing schools. 

Impacts for Repeating Ninth-Grade Students 

While the Talent Development model aims to reduce the number of students who repeat 
the ninth grade by giving intensive support to first-time ninth-grade students, the reform also 
specifically targets students entering the ninth grade for a second time. As explained earlier, 
students who fall into this category have their own team within the Ninth Grade Success Acad- 
emy and work to earn the course credits necessary for midyear promotion; if they do not earn 
midyear promotion, they may obtain additional support in the after-hours Twilight School pro- 
gram. It is difficult to determine the effectiveness of this intervention for all students who begin 
a second year of the ninth grade, because the data only allow for analysis of students who re- 
peated a full year or more. 

For the purposes of this study, repeating ninth-graders are students whose records indi- 
cate that they were in the ninth grade during the spring of the year under study and were also in 
the ninth grade during the previous spring. They did not earn sufficient credits to be promoted to 
the tenth grade on time nor to be promoted midyear. Most of these students were repeating the 
ninth grade for the first time, although some were repeating for the second or even third time. 
Students promoted midyear are classified as tenth-graders in the data and will be included in 
impact analyses for the tenth grade in a future report. If Talent Development’s effort to move 
repeaters to midyear promotion is successful, full-year ninth-grade repeaters left in the subsam- 
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pie are a particularly challenging group of students who have the greatest difficulty meeting 
course requirements. 

Table 7 provides a summary of Talent Development’s impacts for students who re- 
peated the ninth grade for a full year or more. Comparing Table 7 with Table 5 shows that out- 
come levels for repeating ninth-graders were substantially lower than those for first-time ninth- 
graders. Among Talent Development schools, attendance rates in the baseline period for repeat- 
ing ninth-graders averaged 56 percent, compared with approximately 73 percent for first-time 
ninth-graders. Similarly, for Talent Development schools, promotion rates in the baseline period 
for repeating ninth-graders averaged 42 percent, compared with 61 percent for the first-time 
ninth-graders in Talent Development schools. Table 7 also shows that, for most outcomes, base- 
line averages for repeating ninth-graders in the Talent Development schools were even lower 
than those for repeating ninth-graders in the non-Talent Development schools. 

Overall, as Table 7 indicates, Talent Development had little or no impact on most of the 
engagement and performance measures included in the analysis for students who repeated a full 
year or more of ninth grade. An exception is the impact (not statistically significant) on average 
attendance rates, which was actually slightly larger for the repeating ninth-graders (3.0 percent- 
age points) than for the first-time ninth-graders (2.6 percentage points). At the same time, how- 
ever, three outcomes in the table show that, for repeating ninth-graders in Talent Development 
schools, the percentage of students earning four or more credits, the percentage of students earn- 
ing core academic credits, and the percentage of students who were promoted to the tenth grade 
declined slightly, and the declines resulted in negative impacts estimates. However, none of im- 
pact estimates for repeating ninth-graders were large or statistically significant. 



Sizing Up the Results 

Talent Development's intense efforts with ninth-grade students have resulted in positive 
impacts on several student outcomes, including improved attendance rates, increased percent- 
ages of students earning academic credits, and increased promotion rates to the tenth grade. 
First- time ninth-grade students are the greatest beneficiaries of the Talent Development model. 

Impact estimates on academic course-taking and grade-level promotion for first-time 
ninth-graders range between 8 and 19 percentage points, corresponding to effect sizes of be- 
tween 0.15 and 0.41 (effect sizes for attendance outcomes are smaller, ranging between 0.07 
and 0.13). Effect sizes show each impact as a proportion of the district-wide student-level stan- 
dard deviation for each outcome. For example, an increase of 12 percentage points in the per- 
centage of students earning one or more credits in math corresponds to an effect size of 0.23, or 
close to one-quarter, of the district- wide student-level standard deviation for this outcome in the 
pre-Talent Development period. 
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Talent Development Evaluation 
Table 7 



Impacts on Attendance, Course-Taking, and Promotion 
for Repeating Ninth-Grade Students 

Talent Non-Talent 

Development Development 



Outcome 


Baseline 


Schools 

Follow-Up 


Difference 


Baseline 


Schools 

Follow-Up 


Difference 


Impact at 
Follow-Up 


Impact 

Effect 

Size 


Attendance (%) 


Attendance rate 2 


56.0 


60.3 


4.3 ** 


57.2 


58.5 


1.3 


3.0 


0.10 


Students with an attendance rate of: 


90% or higher 


4.8 


6.2 


1.5 


4.7 


5.6 


0.8 


0.6 


0.02 


80% or lower 


87.2 


83.5 


- 3.6 


86.1 


83.4 


- 2.7 * 


-0.9 


-0.02 


Course credits earned (%) 


At least 1 English credit for the year 


35.6 


39.1 


3.6 


39.4 


42.5 


3.2 


0.4 


0.01 


At least 1 algebra credit for the year 


14.6 


21.6 


7 Q ** 


16.7 


20.5 


3.8 ** 


3.2 


0.07 


At least 1 math credit for the year 


29.6 


35.2 


5.6 * 


37.2 


38.3 


1.1 


4.5 


0.09 


4 or more credits for the year 3 


32.6 


32.0 


- 0.6 


33.8 


35.6 


1.8 


-2.4 


-0.05 


Core academic credits 0 


15.3 


13.6 


- 1.7 


17.1 


17.2 


0.1 


-1.9 


-0.04 


Promotion (%) 


Promoted to 10th grade d 


42.1 


39.3 


- 2.8 


45.4 


43.4 


- 2.0 


-0.8 


-0.02 



(continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and seven non-Talent Development high schools. Students in the sample were 
included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school year. Repeating 
9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and were also in the 9th grade in the previous 
year's administrative data file. 
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Table 7 (continued) 



Results in the Talent Development columns reflect averages across the five Talent Development schools. Results in the non-Talent Development columns reflect 
averages across five clusters of non-Talent Development schools. Each cluster consisted of between two and five non-Talent Development schools. Some non-Talent 
Development schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over the three-year period prior to the initial implementation of Talent Development for a given cluster. 

Numbers in the "Follow-Up” columns reflect averages over the entire follow-up period. The length of the follow-up period varies by cluster. Two of the earliest 
implementing clusters had a three-year follow-up period, one cluster had a two-year follow-up period, and two clusters had only a one-year follow-up period. 

Numbers in the “Difference” columns reflect the difference between the baseline average and the average over the follow-up period. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. 

The impact effect size was calculated by dividing the impact at follow-up by the standard deviation of the outcome for all 9th-grade students in the district's 
nonselective, comprehensive high schools from school years 1996-1997 through 1998-1999. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student had 
repeated a prior grade. (For the full 9th-grade sample, the analysis also controls for whether the student was repeating the 9th grade.) 

A two-tailed t-test was applied to the difference between the baseline average and the average over the follow-up period and the impact at follow-up, which is the 
difference in deviations from baseline averages between Talent Development and non-Talent Development schools. Standard errors and statistical significance levels 
are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

“Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a 
given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order to be promoted. Beginning in the 1998-1999 
school year, minimum requirements for promotion included earning at least five credits during the 9th grade, with three of those credits awarded for completing one 
required course in mathematics, one in English, and one in science. 

c “Core academic credits” is a designation indicating that a student completed a core academic curriculum, earning at least five credits during the 9th grade, with 
three of those credits being in mathematics, English, and science. Because this designation relied on the district's course-credit code, which does not distinguish 
between elective courses and required courses, the designation may include credits that students earned for elective courses or for required courses. Some elective 
courses did not meet the district’s new 1998-1999 promotion requirements, but all of them counted toward total credits earned. 

d For the purposes of this analysis, 9th-grade students were considered to have been promoted if they were listed as lOth-graders in the next year's administrative data 
file. Discrepancies between the percentage of students meeting various promotion requirements and the promotion rate may be caused by students earning some 
credits in previous years, incomplete course-detail records, or inconsistent application of the promotion requirements. Due to data availability, this outcome includes 
data for only three of the five clusters of Talent Development and non-Talent Development schools. 




Although no absolute standard exists to define whether a specific effect size is large or 
small, there are some guidelines. For example, many researchers use the rule of thumb that an 
effect size of around 0.20 is small, around 0.50 is moderate, and more than 0.80 is large. 38 This 
categorization was supported empirically by a meta-analysis of more than one hundred treat- 
ment effectiveness studies, most drawn from education research. 39 Most of the benchmarks 
against which to compare effect-size estimates related to elementary school outcomes. For ex- 
ample, the Tennessee class-size experiment, which assessed the impact of reducing elementary 
classrooms from a range of 22 to 26 students to a range of 13 to 17 students, had effects on av- 
erage reading and math test scores of between 0.15 and 0.25 standard deviation. 40 Additionally, 
a meta-analysis of the 29 most widely discussed and disseminated comprehensive school re- 
form models found the average student achievement effect size to be 0.15. This drops to 0.12 
for studies using some sort of comparison group design, and down to 0.09 when a third party 
conducts the evaluation. 41 

Another reference point for the report’s findings are improvements tracked in Chicago 
Public Schools (CPS). In the late- 1990s, Chicago attempted to redesign its high schools with 
goals of increasing academic rigor and enhancing personalization. The design for high schools 
that was adopted by CPS recommended several specific approaches to increased personalization 
that are consistent with Talent Development. The design also focused on a core academic cur- 
riculum for all students and higher graduation standards. A nonexperimental study of CPS 
found that more ninth-grade students were on track to graduate after the high school redesign 
than before. For example, the percentage of eligible ninth-grade students earning enough credits 
to assume sophomore status increased by 5.4 percentage points. The percentage of students 
completing an algebra/geometry sequence rose by 11.1 percentage points. 42 Although these re- 
sults are not part of an evaluation of a specific intervention, they indicate that the magnitude of 
the Talent Development findings reported here is similar to gains made during a sustained dis- 
trict-wide high school reform initiative in another large urban school district. 

Given the above standards, the effect sizes of the impacts for first-time ninth-grade stu- 
dents reported in this study fall in the small-to-moderate categories. However, it is possible for 
small-to-moderate effect sizes to have substantial educational significance. The magnitude of the 
impacts reported here are better than average, compared with other third-party evaluations of 



38 Cohen, 1988; Bloom, 2001. 

39 Lipsey, 1990. 

40 Finn and Achilles, 1999. 

4 'Borman, Hewes, Overman, and Brown, 2003. 

42 Miller and Allensworth, 2002. Quoted values are adjusted for compositional change due to the end of 
social promotion policies. 
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comprehensive school reform models, and they fall in line with other widely hailed reform efforts. 
While the early impact findings in this report should be considered preliminary — as this study 
includes only five high schools in a single district with one to three years of follow-up data — 
they are encouraging. It is not common to find early impacts of this magnitude in evaluations of 
comprehensive school reform models . 43 Overall, the preliminary findings show that Talent Devel- 
opment has positive and significant impacts for keeping ninth-grade students on track for gradua- 
tion, which is one of the most important goals of current high school reform efforts. 

Future Research 

The Talent Development findings show that improvements in the ninth-grade outcomes 
were strongest in the first three schools to begin using the model and were sustained into the 
second and third years of implementation for those schools. While the report describes some 
variation in impacts across the five schools in the study, it does not explore school-specific dif- 
ferences in implementation. Ideally, the analysis should be able to account for the interaction 
between variation in local contexts and Talent Development implementation and variation in 
Talent Development impacts on student engagement and performance. In this case, however, 
the number of dimensions along which this variation was likely to occur far exceeds the number 
of schools and follow-up years included in the analysis. As this work moves forward, the pro- 
gram and evaluation teams will develop more concrete hypotheses about how variation across 
the sites is most likely to affect Talent Development’s capacity to improve student engagement 
and performance. 

Subsequent reports also will track outcomes for up to five follow-up years and will in- 
clude analyses of tenth-, eleventh-, and twelfth-grade students. It will be particularly interesting 
to see whether the impacts continue to be sustained in pioneer schools and whether those im- 
pacts eventually accrue in later-implementing schools. Implementation research will further in- 
vestigate the differences between schools that implement early and those that implement later. 
For some schools, it will be possible to determine whether improvements in ninth-grade promo- 
tion rates translate into increased graduation rates. In addition, it will be interesting to explore 
other measures of student achievement to support the findings related to course credits earned. It 
could be argued that teachers in Talent Development schools have greater incentive to award 
students passing grades that result in increased credits earned, because teachers are part of the 
reform effort and presumably want to see it succeed. At this point, it is not possible to address 
such a hypothesis, but future reports will include analysis of student achievement outcomes. 
Further analysis may also help to unpack the somewhat paradoxical attendance and promotion 



43 Borman, Hewes, Overman, and Brown, 2003; Bloom, 2001. 
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findings, in which there is evidence of substantial improvement in course credits earned and 
promotion to the tenth grade but less evidence of significant changes in attendance rates. One 
hypothesis is that the benefits of Talent Development accrue primarily to students who regularly 
attend school in both the baseline and the follow-up periods. Further study should explore the 
connection between attendance and course completion. 
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PUBLICATIONS FROM MDRC 

Education: School Reform 

A complete publications list is available from MDRC and on its Web site ( www.mdrc.org) , from which 
copies of MDRC's publications can also be downloaded. 



Accelerated Schools 

Research suggests that remediation makes it difficult for elementary school students at risk of 
failure to join the educational mainstream. The Accelerated Schools approach — a whole- 
school reform implemented in more than 1,000 elementary and middle schools — accelerates 
all students' learning. It calls on schools to raise expectations, increase staff involvement in 
decision making, and make curricula more challenging. Completed in 2001, MDRC's evaluation 
measured the reform's effects on student achievement. 

Evaluating the Accelerated Schools Approach 

A Look at Early I mplementation and I mpacts on Student Achievement in Eight 
Elementary Schools 

2001. Howard Bloom, Sandra Ham, Laura Melton, Julienne O'Brien 



Career Academies 

The Career Academy approach to high school reform requires restructuring large schools into 
smaller learning communities, improving instruction, and preparing students for transitions to 
further education or work. Career Academies have been implemented in an estimated 1,500 to 
2,000 high schools, fueling the need for reliable evidence about the approach's affects on 
students' performance and their move from high school. Begun in 1993, MDRC's evaluation of 
Career Academies is the first to use a random assignment research design. 

Career Academies 

Impacts on Labor Market Outcomes and Educational Attainment 
2004. James Kemple, Judith Clayton 

Career Academies 

Impacts on Students' Initial Transitions to Post-Secondary Education and 

Employment 

2001. J ames Kemple 

Career Academies 

Impacts on Students' Engagement and Performance in High School 
2000. James Kemple, Jason Snipes 

Career Academies 

Building Career Awareness and Work-Based Learning Activities Through Employer 
Partnerships 

1999. James Kemple, Susan Poglinco, Jason Snipes 
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Closing Achievement Gaps 



Some of the nation's fastest improving urban school systems are raising overall academic 
performance while reducing achievement gaps among students of different racial groups. But 
instead of taking a school-by-school approach, they are tackling education reform on a 
districtwide basis. Key to their success, according to an exploratory study conducted by MDRC 
for the Council of the Great City Schools, are stable leadership, centrally coordinated changes 
in curriculum and instruction, and data-driven accountability. 

Foundations for Success 

Case Studies of How Urban School Systems Improve Student Achievement 
2002. Jason Snipes, Fred Doolittle, Corinne Herlihy 



Scaling Up First Things First 

MDRC's studies of comprehensive school reforms reflect the depth of its commitment to 
discovering how to improve schools that serve primarily at-risk students. A relatively new and 
ambitious such reform, the First Things First initiative incorporates a variety of best practices 
from real-world settings, including structural changes to create small learning communities in 
large schools, professional development to improve instruction, and higher standards of 
accountability. 

Scaling Up First Things First 

Conducting Classroom Observations in First Things First Schools 
2004. Angela Estacion, Teresa McMahon, Janet Quint 

Scaling Up First Things First 

Findings from the First Implementation Year 

2003. Janet Quint, Crystal Byndloss 

Scaling Up First Things First 
Site Selection and the Planning Year 
2002. J anet Quint 
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About MDRC 



MDRC is a nonprofit, nonpartisan social policy research organization. We are 
dedicated to learning what works to improve the well-being of low-income people. 
Through our research and the active communication of our findings, we seek to 
enhance the effectiveness of social policies and programs. MDRC was founded in 
1974 and is located in New York City and Oakland, California. 

MDRC’s current projects focus on welfare and economic security, education, and 
employment and community initiatives. Complementing our evaluations of a wide 
range of welfare reforms are new studies of supports for the working poor and 
emerging analyses of how programs affect children’s development and their 
families’ well-being. In the field of education, we are testing reforms aimed at 
improving the performance of public schools, especially in urban areas. Finally, our 
community projects are using innovative approaches to increase employment in 
low-income neighborhoods. 

Our projects are a mix of demonstrations — field tests of promising program 
models — and evaluations of government and community initiatives, and we 
employ a wide range of methods to detennine a program’s effects, including large- 
scale studies, surveys, case studies, and ethnographies of individuals and families. 
We share the findings and lessons from our work — including best practices for 
program operators — with a broad audience within the policy and practitioner 
community, as well as the general public and the media. 

Over the past quarter century, MDRC has worked in almost every state, all of the 
nation’s largest cities, and Canada. We conduct our projects in partnership with 
state and local governments, the federal government, public school systems, 
community organizations, and numerous private philanthropies. 
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Unit 1 

Analytic Appendix 




Introduction 

This appendix outlines the analytic approach used by MDRC to estimate the Talent De- 
velopment High School model’s impact on student engagement and performance. Impact find- 
ings are from Talent Development’s first three years of implementation in a large, urban school 
district. The full report, which discusses the findings in detail and describes the components of 
Talent Development and the context in which it operates, is available online at this Web site or 
from MDRC as a printed document. 

Talent Development is a comprehensive reform model for large secondary schools that 
face serious problems with student attendance, discipline, achievement scores, and dropout 
rates. The model calls for specific changes in school organization and curricula with the goals of 
establishing a strong, positive school climate for learning; promoting high standards for English 
and mathematics coursework for all students; and providing professional development systems 
to support implementation of recommended refonns. Each of these changes is aimed specifi- 
cally at enhancing student attendance in school, improving measurable student learning, and 
keeping students on track toward high school graduation. 

The impact analysis for this report focuses on the use of the model in high schools and 
more specifically on engagement and performance outcomes for ninth-grade students. The three 
outcomes that are examined are the ones likely to be most relevant in the early phases of Talent 
Development’s implementation: daily attendance, credits earned, and promotion rates. The ana- 
lytic approach used to measure the model’s impact on these outcomes can best be described as a 
comparative interrupted time series design. 1 

Before detailing the steps of the design, it is important to distinguish between this 
study’s measures of program outcomes and its measures of program impacts. The term “out- 
comes” here refers to the grade level, course credits earned, attendance rate, or behavior of indi- 
vidual students or groups of students at various points during the period under study. The term 
“impacts” here refers to Talent Development’s effects on an outcome. 

Constructing a Counterfactual 

In this study, the average outcome levels (or even year-to-year changes in outcomes) for 
students in the Talent Development schools provide potentially misleading indications of Talent 
Development’s impacts. Previous research has shown that students within a school or set of 



'For further discussion about using an interrupted time series analysis to measure impacts of whole school 
reform, see Bloom (2003) and Snipes (2003). 
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schools may improve from year to year or differ from other students for reasons not necessarily 
related to a special intervention like Talent Development. The ideal research situation would 
allow for an absolutely reliable estimate of the student performance levels that would have been 
observed in the absence of the intervention and that could be compared with actual student per- 
formance — that is, it would allow for a counterfactual. Random assignment is the most reliable 
basis from which to construct estimates of a counterfactual. However, since random assignment 
was not possible for this evaluation, the comparative interrupted time series analysis attempted 
to construct the best counterfactual possible short of random assignment and to estimate the true 
impact of Talent Development. 

To this end, it was necessary to weigh the experiences of a group of students who were 
exposed to Talent Development against those of a comparable group of students who were not. 
The more comparable the two groups prior to the introduction of Talent Development, the more 
likely it is that later differences can be attributed to the program. Moreover, using this kind of 
comparison makes it possible to account for factors other than Talent Development that may 
have caused a change or difference in student engagement and performance. 

The Logic of the Comparative Interrupted Time Series Design 

The comparative interrupted time series design consists of an interrupted time series 
analysis and a comparison school analysis, each of which builds on the strengths of the other 
and addresses the other’s potential limitations. Together, the two analyses construct a counter- 
factual for this evaluation. Specifically, the interrupted time series assesses the extent to which 
measures of engagement and perfomiance for students in Talent Development schools differ 
from measures of engagement and performance for similar students in the same schools prior to 
Talent Development implementation. The school comparison looks at Talent Development 
schools versus non-Talent Development schools (which are similar high schools in the same 
district that are not implementing the reform model). 

The first analysis provides an indication of whether participating high schools experi- 
ence a deviation from their historical patterns in student outcomes coincident with the introduc- 
tion of Talent Development. The projection of each high school’s recent history acts as the 
counterfactual, and it is a particularly good counterfactual because, in the absence of the reform, 
many aspects of the schools would be expected to stay the same (for example, no major changes 
would be expected in students, faculty, school cultures, neighborhoods, or physical plants). Us- 
ing a historical pattern as a counterfactual has the potential to control for both measurable and 
unmeasurable characteristics of a given school. 

However, the deviation from the baseline alone may not necessarily reflect the impact 
of Talent Development. Similar deviations from historical patterns could have been caused by 
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district- wide policies or interventions that occurred at about the same time as Talent Develop- 
ment implementation. For example, while Talent Development scaled up, the schools in the 
district under study changed course promotion requirements . 2 Such a change could cause posi- 
tive deviations from baseline averages of course credits earned at high schools in the district. An 
interrupted time series design would ascribe this improvement to Talent Development, but the 
reform model may have caused some, all, or none of the change in credits earned. 

The second analysis in the comparative interrupted time series design (the comparison be- 
tween Talent Development and Non-Talent Development schools) helps to account for other fac- 
tors in the broader school district that may influence school functioning and student engagement 
and perfonnance. For this part of the analysis, Talent Development schools are matched with sets 
of comparison schools that are similar on several dimensions, including racial composition and 
promotion rates of ninth-grade students, average test scores, and attendance rates. The Talent De- 
velopment and comparison schools are all nonselective, comprehensive high schools in the same 
large, urban district. Measures of student achievement and engagement at the comparison schools 
are a good — but not perfect — indication of what might have been observed in Talent Develop- 
ment schools in the absence of the intervention. 

It should be noted, however, that differences between the Talent Development schools 
in the past and present, and between the Talent Development schools and the comparison 
schools, do not necessarily reflect the actual impact of Talent Development, because some dif- 
ferences could be an artifact of differences in the prior trends in student engagement and per- 
fonnance. For example, test scores for students in Talent Development schools may actually 
have been lower than for students in non-Talent Development schools, and they might have im- 
proved only marginally after Talent Development began. At the same time, test scores for stu- 
dents attending similar non-Talent Development schools in the district may actually have been 
declining over the same period. In such an instance, Talent Development would have a positive 
impact by preventing test scores from dropping rather than by improving the overall average. 
This could be observed only by comparing an interrupted time series for both Talent Develop- 
ment and non-Talent Development comparison schools. 

The comparative interrupted time series analysis makes this comparison by estimating 
the deviations from historical patterns for the Talent Development schools and subtracting from 
these the deviations from historical patterns for similar non-Talent Development schools during 
the same period. The differences between these deviations constitute impacts in student out- 
comes. When combined with regression analysis to control for differences in individual student 



‘In order to preserve the anonymity of the subjects in this study, this appendix refers to the participating 
school district as “the district” and uses pseudonyms for individual schools. 



- 3 - 




background characteristics and prior school experiences, the approach isolates the unique im- 
pact Talent Development has on student engagement and performance. 

It should be noted, however, that the comparative interrupted time series approach carries 
with it the limitations that are present in all quasi-experimental designs. In this case, projection of 
a baseline average for a given school may not be a reliable predictor of future student outcomes. 
Also, finding comparison schools for the Talent Development schools is limited to observable 
characteristics of the student body and may miss important factors that affect student outcome 
trajectories. In addition, multiple-regression techniques control for compositional changes in 
measurable student characteristics, but there may be changes in unmeasurable student characteris- 
tics that correlate with student outcomes. Despite these limitations, using the comparative inter- 
rupted time series approach to pool impacts estimates over several schools provides the best esti- 
mate of the true impact of Talent Development in high schools in the district. 

The primary sources of data for this analysis were individual students’ school records, 
which were obtained from the district. Table 1 provides a list of the types of data that were ob- 
tained and the school years and grade levels for which they were available. In general, adminis- 
trative, attendance, and course-detail infomiation is available for all high school students in the 
district from the 1995-1996 school year through the 2001-2002 school year. Table 2 gives a 
brief overview of the types of information included in these data sets. Table 3 defines several 
key outcomes included in the analysis. 

The rest of this appendix will provide a step-by-step description of the analyses. The 
following section details the steps that make up the interrupted time series approach, including 
estimating deviations from baseline, controlling for compositional shifts, and accounting for 
cohort effects. The section after that describes the comparison school approach, including se- 
lecting comparison schools and estimating their deviations from baseline. The final section de- 
scribes estimating impacts and pooling estimates across schools. To provide a concrete example 
in support of the descriptions, this appendix refers throughout to the ninth-grade outcome of 
earning one or more math credits. 



Interrupted Time Series Approach 

Estimating Deviations from Baseline 

For this evaluation, outcomes for students enrolled in a given school prior to Talent De- 
velopment’s implementation were compared with outcomes for students enrolled in the same 
school during the years after implementation began. For most measures of student engagement 
and performance, the analysis focuses on the three years prior to implementation and the three 
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Talent Development Evaluation 

Analytic Appendix 
Table 1 

Data Sources and Availability, 
by School Year and Grade Level 



School Y ear 



Data Source 


1991-1992 


1992-1993 


1993-1994 


1994-1995 


1995-1996 


1996-1997 


1997-1998 


1998-1999 


1999-2000 


2000-2001 


2001-2002 


Administrative records 










9-12 


6-12 


6-12 


6-12 


6-12 


6-12 


6-12 


Attendance records 










9-12 


6-8 


6-12 


6-12 


6-12° 


6-12 a 


6-12 


Course-detail records 










6-12 


6-12 


6-12 


6-12 


6-12 


6-12 


6-12 


Test scores 
























CTBS 


1-8 


1-8 


1-8 


4-8 b 
















SAT-9 










2-4, 6-8, 1 1 


2-4, 6-8, 1 1 


2-4, 7-8 


2-4, 7-8 


2-4, 7-8, 

10-11 


3, 4, 7, 8 


3,4, 7, 10 


SSA 










5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 



SOURCE: Individual students' school records from a large, urban school district. 

NOTES: Blank spaces indicate that no records are available for those years. 

Administrative records include information on students' race; gender; birth date; and final school-enrollment status for the year, including withdrawal and dropout status and 
number of suspensions. 

Attendance records include information on the number of days a student is present and absent for each marking period. Unless otherwise noted, this sample includes students 
who attended at least one day in any of the marking periods. 

Course-detail records include information on credits attempted, credits earned, grades, and absences for each course in which a student was enrolled during the year. Unless 
otherwise noted, this sample includes students who were enrolled in at least one course during the year, according to the course-detail records. 

Test scores may not be available for every student. 

Comprehensive Test of Basic Skills (CTBS) records include test scores for reading, math, science, and social studies. 

Stanford Achievement Test, Ninth Edition (SAT-9) records include test scores for reading, math, problem solving, procedures, and science. 

State Standards Assessment (SSA) records include test scores for reading and math. 

a Attendance records for years 1 999-2000 and 2000-200 1 include only students who were present for at least one day in the last marking period. 

b Test scores for 1994-1995 are missing for a number of middle and high schools. 
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Analytic Appendix 
Table 2 



Data Types and Descriptions 



Data Type 


Description 


1 1 


Administrative 


Administrative data typically include student background information, such as birth date, race, 
and gender, as well as information on school enrollment status, special education classification, 
and English Speakers of Other Languages (ESOL) training. The administrative data are also the 
primary source of information about the grade level in which students were enrolled during each 
school year. These records were used to determine whether students were promoted from year to 
year or retained in grade. Typically, administrative records are available for all students in a 
given school level regardless of whether they entered the district after the school year began or 
whether they dropped out or left the district before the end of the year. 


1 1 


Attendance 


Attendance data include information about the number of days a student was present or absent 
during a given school year. In some years, these data were provided on a quarterly basis and in 
other years they were provided as cumulative records. This information was used to construct an 
attendance rate and an absentee rate for each student in the files. Typically, the attendance files 
include only students who were present for at least one day during the final making period of the 
year. This means that students who dropped out of school or who left the district before the start 
of the final marking period do not have an attendance record for this analysis. 


1 1 


Course-detail 

records 


Course-detail records include, for each course in which a student was enrolled during a given 
school year, the course code number, an abbreviated name, the number of credits the student 
attempted, the number of credits the student earned, and the grade the student received. For each 
student in the file, this information was used to construct both an annual and a cumulative count 
of credits earned and attempted. The information was also used to calculate credits earned in 
particular subject areas. 


1 1 


Test scores 
(nationally 
normed) 


The California Test of Basic Skills (CTBS) and the Stanford Achievement Test, Ninth Edition 
(SAT -9) are norm-referenced test scores, which provide infomiation on individual student 
achievement relative to scores obtained from a random sample of students from across the 
country. SAT-9 scores in math and reading are available as Normal Curve Equivalents, National 
Percentiles, and Scale Scores. In general, these test scores were used in the analysis to control 
for student achievement prior to entering high school. 


1 1 


Test scores 
(state) 


The State Standards Assessment (SSA) is a criterion-referenced test, which provides information 
on student skills and content knowledge specified by the state. SSA test scores in math and 
reading are available as Normal Curve Equivalents (NCEs), State Percentiles, and Scale Scores 
for each of these grades and the school years listed in Analytic Appendix Table 1 . 
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Analytic Appendix 
Table 3 

Key Outcome Definitions 



Outcome 


Definition 


1 1 


Attendance rate 


The total number of days a student was marked present during the school year divided by the 
total number of days the student was listed as enrolled. These data were available consistently 
only for students who attended school for at least one day in the final marking period of the 
year. Thus, the analysis did not include students who dropped out or left the district prior to that 
point. 


1 1 


Chronic 

absenteeism 


Indicates that a student had an attendance rate of 80 percent or lower for the year. 


1 1 


Regular 

attendance 


Indicates that a student had an attendance rate of 90 percent or higher for the year. 


1 1 


Course-taking 

patterns 


Indicators of whether a high school student earned course credits in selected subjects, including 
mathematics, English, and science. Other indicators, which signify credits earned in academic 
subjects, career-technical subjects, and other electives, are also available. 


1 1 


Earned 4 or more 
credits 


Designation given to a high school student who was awarded credit for having passed 4 or more 
courses for the year. Until the 1998-1999 school year, students were required to earn 4 or more 
credits in order to be promoted to the next grade. 


1 1 


Met minimum 
requirements for 
promotion 


Beginning in the 1998-1999 school year, minimum requirements for promotion included 
earning at least 5 credits during the year, with at least one of these credits in English, one in 
mathematics, and one in science. 


1 1 


Promoted to the 
9th grade on time 


Designation given to a student who was listed as a 9th-grader in a given school year in the 
district’s administrative records and as a lOth-grader in the following school year. Note that this 
is not necessarily the same as indicating that a student met various criteria for promotion, such 
as earning the required number of course credits or completing a designated sequence. While 
the vast majority of students who were promoted according to the definition above did in fact 
meet these requirements, some students were promoted without meeting them. Some of these 
students may have completed some of the necessary requirements in a prior year. 
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years after implementation. The three years prior to implementation are referred to as the base- 
line period. The year of implementation and each subsequent year are referred to as follow-up 
years. Differences in student outcomes between the baseline and follow-up periods are referred 
to as deviations from the baseline. 

The key feature of the interrupted time series design is to project what student engage- 
ment and perfonnance would most likely be without Talent Development. This projection ex- 
tends over one or more years after Talent Development began and is based on measures of stu- 
dent engagement and perfomiance during a multiyear pre-Talent Development baseline period. 
For example, to project into the follow-up period students’ pattern of earning one or more math 
credits for a given year, the analysis used the annual percentage of ninth-graders earning one or 
more math credits over the three baseline years. The equation below specifies the simplest fonn 
of a regression model that can be used to estimate an interrupted time series from a baseline de- 
rived from the three-year average at a single school. 3 

Y, = A + j^D,FY„+e, 

k = 1 

where: 

T = dummy for earning one or more math credits for student i 

FY kj = 1 if student i was a member of the cohort for follow-up year k, 

and 0 otherwise 

e i = a random error tenn for student i 

A = a constant term equal to the percentage of ninth-grade students 

earning one or more math credits during the baseline years 

D k = deviation in the percentage of ninth-grade students earning one 

or more math credits from the baseline average A in year k 
of the follow-up period (that is, the year k deviation 
from the baseline mean) 

This equation pools data from the baseline and follow-up years and estimates both the 
baseline mean and the average deviation from this mean for each year of the follow-up period 



3 It is also possible to project a baseline trend derived from a consistent pattern of year-to-year increases or 
decreases in credits earned in the pre-Talent Development period. This was discounted for the current analysis 
because only three years of pre-Talent Development data were available, leaving only minimal confidence in 
an estimate of a consistent year-to-year slope in baseline credits-eamed patterns. See Bloom (2003) for both 
baseline-trend and baseline-average interrupted time series techniques. 
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Analytic Appendix 
Figure 1 

Percentage of Ninth-Grade Students Earning One or More Math Credits for the Year 

in Talent Development School A, 

Three-Year, Unadjusted, Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The sample includes all students whose course-detail records indicate that they attempted one or more credits. 



for a single school. Figure 1 shows the unadjusted interrupted time series estimates for one Tal- 
ent Development school in the district, School A. The triangles plot observed means for each 
baseline year. The solid line represents the baseline average, while the dashed line is the projec- 
tion of this average into the first three Talent Development implementation years. The circles 
plot observed means for each follow-up year. The difference between the dashed line and each 
circle represents the deviation from baseline average for each year of implementation. (Note 
that the years listed in the x-axis of this figure and Figures 2, 3, 4, and 5 are presented relative to 
the first year of Talent Development implementation.) 

Controlling for Changes in Student Characteristics 

In some cases, a Talent Development school (or a comparison school) may experience a 
change in the composition of its student population. For example, a neighborhood may undergo 
demographic changes, or geographic boundaries or rules governing school assigmnent patterns 
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may change. More importantly, Talent Development may cause a change in the student popula- 
tion, by preventing students from dropping out of high school, for example (which may keep 
lower-performing students in school longer). In order to help account for systematic changes in 
the characteristics of student cohorts over time, the analysis incorporates individual student 
characteristics into the model. The following equation represents the enhanced regression model 
for a single school: 

r, = A + i,D t FY kl +£ J C J X f +e, 

k= 1 7=1 

where the parameters specified above are the same and: 

X /; = a vector of J background characteristics for student i 

C j = the difference in the percentage of ninth-graders earning one or more 

math credits over time associated with a unit change in background 
characteristics X 

The capacity of the analysis to control for systematic changes in the characteristics of 
student cohorts is increased if the X covariates and the outcomes are correlated. For example, 
suppose that, in one school, Talent Development increases the percentage of ninth-graders who 
had been retained in a prior grade or who start high school with very low achievement levels. 
Such a scenario might occur if Talent Development prevents these students from dropping out 
of school altogether. Because such students are likely to be retained in grade and less likely to 
complete course requirements, it could appear that Talent Development is reducing average 
course credits earned if the analysis does not account for the change in the composition of the 
ninth grade. Thus, it will be important to identify characteristics that are correlated with key out- 
comes, such as retention in grade. This can help disentangle Talent Development’s impact on 
earning course credits from impacts that are caused by changes in the composition of the ninth- 
grade cohorts. In this case, the following covariates were incorporated into the interrupted times 
series models: 

Overage = whether the student was overage for the current grade, 
indicating that she or he had been retained in a previous 
grade 

Race = dummy variables indicating whether the student was 
black, white, or of another race 

Test scores = Separate variables indicating students’ eighth-grade reading 
comprehension and math test scores (in normal curve 
equivalents) 
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Figure 2 shows the adjusted interrupted time series estimates for Talent Development 
School A. As in Figure 1, the triangles plot observed means for each baseline year. The solid 
line represents the baseline average, while the dashed line is the projection of this average into 
the first three Talent Development implementation years. The circles plot observed means for 
each follow-up year. The difference between the dashed line and each circle represents the de- 
viation from baseline average for each year of implementation. 
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Figure 2 

Percentage of Ninth-Grade Students Earning One or More Math Credits for the Year 

in Talent Development School A, 

Three- Year, Adjusted, Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The sample includes all students whose course-detail records indicate that they attempted one or more credits. 



Accounting for Cohort Effects 

In addition to controlling for changes in student characteristics, the analysis also at- 
tempts to account for cohort effects which are year-to-year variations in the average engage- 
ment and perfonnance of students as a group. Because cohort effects reflect variation that can- 
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